Oct 30 00:01:55.975189 ip-10-0-137-69 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 00:01:55.975201 ip-10-0-137-69 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 00:01:55.975209 ip-10-0-137-69 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 00:01:55.975518 ip-10-0-137-69 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 00:02:07.341624 ip-10-0-137-69 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 00:02:07.341641 ip-10-0-137-69 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bcda903527744980bbc4123d4db62cd1 --
Oct 30 00:04:10.877283 ip-10-0-137-69 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:04:11.215188 ip-10-0-137-69 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:04:11.215188 ip-10-0-137-69 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:04:11.215188 ip-10-0-137-69 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:04:11.215730 ip-10-0-137-69 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:04:11.218926 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218855    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:11.218926 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218926    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218930    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218933    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218936    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218939    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218941    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218944    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218947    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218950    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218952    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218955    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218958    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218961    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218964    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218966    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218969    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218971    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218974    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218977    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218979    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:11.218987 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218982    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218984    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218987    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218990    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218993    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.218995    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219004    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219007    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219010    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219012    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219015    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219018    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219020    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219035    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219039    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219044    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219047    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219050    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219052    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:11.219475 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219055    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219057    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219060    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219062    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219065    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219067    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219070    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219072    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219075    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219078    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219080    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219083    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219085    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219088    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219090    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219094    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219096    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219099    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219102    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219104    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:11.219949 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219108    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219112    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219116    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219118    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219121    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219123    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219126    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219128    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219131    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219133    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219136    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219138    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219141    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219143    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219146    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219148    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219151    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219153    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219156    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:11.220431 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219160    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219162    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219165    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219168    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219170    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219173    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219175    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219178    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219181    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219184    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219186    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219189    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219191    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219194    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219197    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219199    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219202    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219205    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219207    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219210    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:11.220872 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219213    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219215    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219218    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219221    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219224    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219226    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219229    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219960    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219967    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219970    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219973    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219975    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219979    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219982    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219984    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219987    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219990    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219993    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219995    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.219998    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:11.221355 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220001    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220003    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220006    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220009    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220012    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220014    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220017    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220020    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220050    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220054    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220057    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220059    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220062    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220065    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220068    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220070    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220074    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220079    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220082    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:11.221913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220085    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220087    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220090    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220092    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220096    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220101    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220104    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220108    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220111    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220114    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220117    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220120    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220123    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220125    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220128    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220131    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220133    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220136    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220139    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:11.222512 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220141    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220144    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220148    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220150    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220153    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220155    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220158    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220160    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220163    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220165    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220168    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220170    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220173    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220175    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220178    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220180    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220183    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220186    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220188    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220191    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:11.223206 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220193    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220196    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220198    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220201    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220203    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220206    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220210    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220212    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220215    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220218    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220220    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220223    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220225    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220228    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220230    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220233    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220236    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220238    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220241    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:11.223717 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220243    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220245    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220248    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220250    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220253    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220256    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220258    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220261    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220263    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220265    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220268    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220271    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220274    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220276    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220279    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220281    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220786    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220793    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220796    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220799    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:11.224180 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220802    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220804    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220809    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220813    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220816    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220819    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220822    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220825    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220828    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220831    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220833    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220836    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220838    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220841    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220843    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220847    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220849    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220852    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220854    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:11.224676 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220857    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220859    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220863    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220865    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220868    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220870    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220873    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220876    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220879    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220881    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220884    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220887    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220890    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220895    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220898    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220901    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220904    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220906    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220909    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:11.225137 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220912    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220914    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220917    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220925    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220928    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220931    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220933    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220936    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220938    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220941    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220944    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220947    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220949    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220952    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220954    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220957    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220959    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220962    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220965    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220967    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:11.225580 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220970    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220972    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220975    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220977    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220980    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220982    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220985    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220988    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220990    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220993    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220996    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.220998    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221001    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221003    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221005    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221008    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221016    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221019    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221036    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:11.226090 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221039    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221042    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221045    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221047    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221050    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221052    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221055    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221057    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221060    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221063    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221065    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221068    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221070    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221073    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221075    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221078    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221081    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221084    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221086    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221089    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:11.226538 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221091    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221093    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221096    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221098    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.221101    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222037    2581 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222048    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222054    2581 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222059    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222063    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222112    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222783    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222792    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222798    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222803    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222809    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222816    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222821    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222827    2581 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222837    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222842    2581 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222847    2581 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222852    2581 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:04:11.227046 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222883    2581 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222887    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222892    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222897    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222903    2581 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222915    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222921    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222926    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222932    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222937    2581 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222941    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222947    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222952    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222961    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222968    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222973    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222978    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222983    2581 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222988    2581 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.222993    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223178    2581 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223206    2581 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223330    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223337    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:04:11.227588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223343    2581 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223351    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223356    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223361    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223367    2581 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223371    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223376    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223382    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223386    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223391    2581 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223396    2581 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223409    2581 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223416    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223421    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223427    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223432    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223436    2581 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223441    2581 flags.go:64] FLAG: --help="false"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223447    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223452    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223483    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223490    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223496    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:04:11.228180 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223502    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223507    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223511    2581 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223516    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223520    2581 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223524    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223527    2581 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223530    2581 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223534    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223537    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223540    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223543    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223545    2581 flags.go:64] FLAG: --lock-file=""
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223548    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223551    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223555    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223561    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223564    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223567    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223569    2581 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223572    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223576    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223579    2581 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223583    2581 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223588    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:04:11.228722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223591    2581 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223595    2581 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223598    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223601    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223605    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223608    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223611    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223614    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223617    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223632    2581 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223636    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223639    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223642    2581 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223644    2581 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223647    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223650    2581 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223653    2581 flags.go:64] FLAG: --port="10250"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223656    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223659    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02d2468df5f0482ca"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223663    2581 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223666    2581 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223669    2581 flags.go:64] FLAG: --register-node="true"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223671    2581 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223675    2581 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223678    2581 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223681    2581 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:04:11.229413 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223684    2581 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223688    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223691    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223695    2581 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223698    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223701    2581 flags.go:64] FLAG: --runonce="false"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223704    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223707    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223710    2581 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223713    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223716    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223719    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223722    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223726    2581 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223729    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223732    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223735    2581 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223738    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223741    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223744    2581 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223747    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223752    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223755    2581 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223758    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223763    2581 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:04:11.230540 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223766    2581 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223769    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223772    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223775    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223777    2581 flags.go:64] FLAG: --v="2"
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223786    2581 flags.go:64] FLAG: --version="false"
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223791    2581 flags.go:64] FLAG: --vmodule=""
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223795    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.223799    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.223959    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.223963    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.223967    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.223972    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.223980    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.223984    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.223986    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.223989    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.223992    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.223994    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.223997    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224000    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:11.231317 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224002    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224005    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224008    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224011    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224014    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224016    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224019    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224035    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224038    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224041    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224044    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224047    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224050    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224053    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224055    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224058    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224060    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224063    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224065    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:11.232209 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224068    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224073    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224078    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224081    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224084    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224087    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224097    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224100    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224103    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224105    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224108    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224110    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224113    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224115    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224118    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224120    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224123    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224125    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224128    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224130    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:11.233015 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224133    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224136    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224138    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224141    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224144    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224146    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224149    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224151    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224154    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224156    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224159    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224161    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224164    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224167    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224170    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224172    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224175    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224178    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224180    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:11.233956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224189    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224192    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224194    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224197    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224199    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224202    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224204    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224207    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224209    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224212    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224214    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224217    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224219    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224221    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224224    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224227    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224229    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224231    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224234    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:11.234637 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224237    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224239    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224242    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224244    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224246    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224249    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224251    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224254    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224258    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224262    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224265    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224268    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224271    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224273    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224282    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224284    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:11.235302 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224287    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.224316    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224353    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224358    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224362    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224366    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224370    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224373    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224376    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224379    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224381    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224384    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224387    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224390    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224392    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224394    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:11.235971 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224397    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224400    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224403    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224405    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224408    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224410    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224413    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224415    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224418    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224421    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224423    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224426    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224428    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224431    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224434    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224442    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224445    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224447    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224450    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224453    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:11.236508 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224455    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224458    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224461    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224463    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224465    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224468    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224470    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224473    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224475    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224478    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224480    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224483    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224487    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224490    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224492    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224495    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224497    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224500    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224502    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:11.237231 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224505    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224510    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224513    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224516    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224518    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224521    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224523    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224526    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224528    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224537    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224540    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224542    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224545    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224547    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224551    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224554    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224558    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224560    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224563    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:11.237686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224566    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224568    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224571    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224573    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224575    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224578    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224582    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224584    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224587    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224590    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224592    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224595    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224597    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224600    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224603    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224606    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224609    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224612    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224614    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224617    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:11.238152 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224619    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224622    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224625    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224633    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224636    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224638    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224641    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224644    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224646    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224649    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224652    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224654    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224657    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.224659    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.232240    2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.232260    2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.232287    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.232295    2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232378    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232385    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:11.238619 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232391    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232396    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232401    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232405    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232409    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232413    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232420    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232426    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232432    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232437    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232441    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232445    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232451    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232455    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232459    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232464    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232468    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232472    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232477    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:11.239150 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232481    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232486    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232490    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232494    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232499    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232503    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232507    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232511    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232520    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232528    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232533    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232538    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232542    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232547    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232552    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232556    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232560    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232564    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232569    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:11.239626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232573    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232577    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232581    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232585    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232590    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232594    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232598    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232603    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232607    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232612    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232616    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232620    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232624    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232628    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232633    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232637    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232641    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232645    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232649    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:11.240097 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232653    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232658    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232663    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232668    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232672    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232676    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232681    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232685    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232689    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232693    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232697    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232701    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232705    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232709    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232713    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232717    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232722    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232726    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232731    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232735    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:11.240544 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232740    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232744    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232748    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232752    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232756    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232761    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232765    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232769    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232773    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232777    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232781    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232785    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232790    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232794    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232799    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232803    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232808    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232812    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232816    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232820    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:11.241119 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232825    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232829    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232833    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232837    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232842    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232846    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.232850    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.232892    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233133    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233143    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233148    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233152    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233156    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233160    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233165    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233169    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:11.241634 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233173    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233177    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233182    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233186    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233194    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233201    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233207    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233212    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233216    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233221    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233225    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233229    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233234    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233239    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233244    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233248    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233252    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233256    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233260    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:11.242009 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233265    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233269    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233274    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233278    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233282    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233287    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233291    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233295    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233299    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233303    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233308    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233313    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233317    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233321    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233325    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233329    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233334    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233338    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233342    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233346    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:11.242472 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233350    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233354    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233359    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233363    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233367    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233371    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233376    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233381    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233385    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233391    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233395    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233399    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233403    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233407    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233411    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233428    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233433    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233438    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233442    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233446    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:11.242956 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233450    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233454    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233458    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233463    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233467    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233471    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233475    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233480    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233484    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233488    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233492    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233496    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233500    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233504    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233509    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233513    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233518    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233524    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233530    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:11.243438 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233535    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233541    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233546    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233550    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233555    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233559    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233564    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233568    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233572    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233576    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233581    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233585    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233589    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233593    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233598    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233603    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233607    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233611    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233615    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:11.243868 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233620    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.233661    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233712    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233720    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233725    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233730    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233734    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233738    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233743    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233747    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233752    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233756    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233760    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233764    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233769    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233781    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:11.244323 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233785    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233790    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233794    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233798    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233802    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233807    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233811    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233815    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233819    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233826    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233831    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233835    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233839    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233844    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233848    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233854    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233858    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233863    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233867    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233871    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:11.244686 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233875    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233880    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233884    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233888    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233893    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233897    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233902    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233906    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233911    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233915    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233920    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233924    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233929    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233933    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233937    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233941    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233946    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233950    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233954    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:11.245163 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233958    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233962    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233968    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233972    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233976    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233980    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233984    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233989    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233995    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.233999    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234003    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234008    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234012    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234016    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234020    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234039    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234044    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234050    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234056    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:11.245626 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234060    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234066    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234071    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234075    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234080    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234084    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234088    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234093    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234098    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234102    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234108    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234112    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234116    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234121    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234125    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234131    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234136    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234140    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234144    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234149    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:11.246076 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234154    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234160    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234164    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234169    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234173    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234178    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234182    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234187    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234191    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234195    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234199    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234204    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234210    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:11.234217    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.234326    2581 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.236892    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.237820    2581 server.go:1014] "Starting client certificate rotation"
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.237920    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:04:11.246529 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.237948    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:04:11.246975 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.239647    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:04:11.246975 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.243695    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:04:11.255574 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.255555    2581 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:04:11.258943 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.258930    2581 log.go:25] "Validated CRI v1 image API"
Oct 30 00:04:11.259761 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.259739    2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:04:11.262502 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.262478    2581 fs.go:135] Filesystem UUIDs: map[12b6fd48-b32b-4065-a06c-c11ead05fb55:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 eee7c887-bd34-4bff-a0e3-52bb7a2cd80a:/dev/nvme0n1p3]
Oct 30 00:04:11.262562 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.262501    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 00:04:11.267753 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.267641    2581 manager.go:217] Machine: {Timestamp:2026-10-30 00:04:11.266448396 +0000 UTC m=+0.290148442 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108024 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20d383c00d4a614a6b4b705de3ef75 SystemUUID:ec20d383-c00d-4a61-4a6b-4b705de3ef75 BootID:bcda9035-2774-4980-bbc4-123d4db62cd1 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c5:cc:2a:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c5:cc:2a:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:f2:6e:4d:97:b6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:04:11.267753 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.267753    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:04:11.267858 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.267827    2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:04:11.270300 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.270279    2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:04:11.270458 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.270301    2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-69.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:04:11.270551 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.270464    2581 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:04:11.270551 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.270477    2581 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:04:11.270551 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.270525    2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:04:11.270551 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.270548    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:04:11.271386 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.271371    2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:04:11.274035 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.274013    2581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:04:11.274152 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.274142    2581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:04:11.274665 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.274655    2581 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:04:11.274694 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.274668    2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:04:11.274694 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.274678    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:04:11.274694 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.274688    2581 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:04:11.274801 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.274695    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:04:11.275075 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.275056    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:04:11.276837 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.276809    2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:04:11.281311 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.281295    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:04:11.281960 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.281945    2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:04:11.282011 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.281976    2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:04:11.282164 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.282151    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:04:11.282203 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.282174    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:04:11.282203 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.282186    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:04:11.282203 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.282197    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:04:11.282282 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.282209    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:04:11.282282 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.282222    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:04:11.283104 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.283094    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:04:11.283104 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.283104    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:04:11.283169 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.283112    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:04:11.283169 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.283118    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:04:11.283169 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.283127    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:04:11.283169 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.283135    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:04:11.283869 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.283859    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:04:11.283917 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.283870    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:04:11.287423 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.287394    2581 server.go:1263] "Started kubelet"
Oct 30 00:04:11.287505 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.287456    2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:04:11.287652 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.287510    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:04:11.287722 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.287711    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:04:11.288058 ip-10-0-137-69 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:04:11.289686 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.289596    2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:04:11.289686 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.289627    2581 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:04:11.295190 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:11.295170    2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 00:04:11.296581 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.296567    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:04:11.296913 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.296894    2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:04:11.297070 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:11.297052    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-69.ec2.internal\" not found"
Oct 30 00:04:11.297686 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.297673    2581 factory.go:55] Registering systemd factory
Oct 30 00:04:11.297686 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.297687    2581 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:04:11.297819 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.297718    2581 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:04:11.297819 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.297733    2581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:04:11.297819 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.297716    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:04:11.297912 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.297824    2581 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:04:11.297912 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.297830    2581 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:04:11.297912 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.297899    2581 factory.go:153] Registering CRI-O factory
Oct 30 00:04:11.297912 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.297907    2581 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:04:11.298020 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.297944    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:04:11.298020 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.297957    2581 factory.go:103] Registering Raw factory
Oct 30 00:04:11.298020 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.297966    2581 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:04:11.298709 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.298694    2581 manager.go:319] Starting recovery of all containers
Oct 30 00:04:11.307870 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.307854    2581 manager.go:324] Recovery completed
Oct 30 00:04:11.308516 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.308466    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:04:11.313077 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.313064    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:04:11.315033 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.315003    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:04:11.315099 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.315063    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:04:11.315099 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.315076    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:04:11.315607 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.315593    2581 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:04:11.315607 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.315605    2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:04:11.315723 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.315624    2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:04:11.315723 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.315628    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:04:11.316408 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.316389    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:04:11.316471 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.316423    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:04:11.316471 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.316445    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:04:11.316960 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.316942    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:04:11.317043 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.317002    2581 policy_none.go:50] "Start"
Oct 30 00:04:11.317043 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.317015    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:04:11.317043 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.317038    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:04:11.317990 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.317980    2581 policy_none.go:44] "Start"
Oct 30 00:04:11.330196 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.329707    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-69.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 00:04:11.333219 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:11.329703    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-69.ec2.internal.18e325cfdb10d6b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-69.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-69.ec2.internal,},FirstTimestamp:2026-10-30 00:04:11.287361203 +0000 UTC m=+0.311061248,LastTimestamp:2026-10-30 00:04:11.287361203 +0000 UTC m=+0.311061248,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-69.ec2.internal,}"
Oct 30 00:04:11.333627 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.333605    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-874ld"
Oct 30 00:04:11.338824 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:11.338805    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-69.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 00:04:11.342631 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.342614    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-874ld"
Oct 30 00:04:11.343723 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.343699    2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:04:11.346170 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.346157    2581 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:04:11.346243 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:11.346184    2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:04:11.346243 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.346196    2581 server.go:87] "Starting device plugin registration server"
Oct 30 00:04:11.346420 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.346409    2581 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:04:11.346462 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.346424    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:04:11.346572 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.346554    2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:04:11.346648 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.346637    2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:04:11.346648 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.346648    2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:04:11.347116 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:11.347095    2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:04:11.347202 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:11.347146    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-69.ec2.internal\" not found"
Oct 30 00:04:11.347715 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:11.347641    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-69.ec2.internal.18e325cfdcb6e2ef  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-69.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-69.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-69.ec2.internal,},FirstTimestamp:2026-10-30 00:04:11.315020527 +0000 UTC m=+0.338720577,LastTimestamp:2026-10-30 00:04:11.315020527 +0000 UTC m=+0.338720577,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-69.ec2.internal,}"
Oct 30 00:04:11.347828 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:11.347748    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 00:04:11.347902 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:11.347865    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 00:04:11.376719 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.376701    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:04:11.434955 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.434937    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:04:11.435097 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.434959    2581 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:04:11.435097 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.434974    2581 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:04:11.435097 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:11.435001    2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 00:04:11.446746 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.446728    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:04:11.447499 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.447480    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:04:11.447586 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.447512    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:04:11.447586 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.447531    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:04:11.447586 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.447552    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.449264 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.449250    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:04:11.464373 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.464361    2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.464415 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:11.464383    2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-69.ec2.internal\": node \"ip-10-0-137-69.ec2.internal\" not found"
Oct 30 00:04:11.535946 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.535790    2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"]
Oct 30 00:04:11.537515 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.537502    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.538299 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.538284    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.571512 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.571495    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.575077 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.575062    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.595981 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.595957    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:04:11.598633 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.598619    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:04:11.599735 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.599719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/689ea6de40d51a70fe308b471157700c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" (UID: \"689ea6de40d51a70fe308b471157700c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.599787 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.599749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/689ea6de40d51a70fe308b471157700c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" (UID: \"689ea6de40d51a70fe308b471157700c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.599787 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.599765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3314e96d9e96380a527867d39d2fde5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.ec2.internal\" (UID: \"e3314e96d9e96380a527867d39d2fde5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.700202 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.700177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/689ea6de40d51a70fe308b471157700c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" (UID: \"689ea6de40d51a70fe308b471157700c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.700202 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.700203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/689ea6de40d51a70fe308b471157700c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" (UID: \"689ea6de40d51a70fe308b471157700c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.700353 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.700225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3314e96d9e96380a527867d39d2fde5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.ec2.internal\" (UID: \"e3314e96d9e96380a527867d39d2fde5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.700353 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.700270    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3314e96d9e96380a527867d39d2fde5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.ec2.internal\" (UID: \"e3314e96d9e96380a527867d39d2fde5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.700353 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.700292    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/689ea6de40d51a70fe308b471157700c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" (UID: \"689ea6de40d51a70fe308b471157700c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.700353 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.700291    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/689ea6de40d51a70fe308b471157700c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" (UID: \"689ea6de40d51a70fe308b471157700c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.900593 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.900534    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:11.902697 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:11.902679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:12.237836 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.237765    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 00:04:12.238272 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.237951    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 00:04:12.275824 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.275802    2581 apiserver.go:52] "Watching apiserver"
Oct 30 00:04:12.297789 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.297760    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 00:04:12.346321 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.346282    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 23:59:11 +0000 UTC" deadline="2028-03-27 12:34:45.990512274 +0000 UTC"
Oct 30 00:04:12.346321 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.346318    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12348h30m33.644197332s"
Oct 30 00:04:12.348251 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.348227    2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:04:12.348600 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.348581    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-sf6zz","kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal","openshift-network-operator/iptables-alerter-hnp8z","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k","openshift-cluster-node-tuning-operator/tuned-wtbxs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal","openshift-multus/multus-additional-cni-plugins-8k6ms","openshift-multus/network-metrics-daemon-6cnnk","kube-system/konnectivity-agent-kmg9c","openshift-dns/node-resolver-jvrpq","openshift-multus/multus-rrphh","openshift-network-diagnostics/network-check-target-tr4w8","kube-system/global-pull-secret-syncer-v5tjs","openshift-image-registry/node-ca-6hq6g"]
Oct 30 00:04:12.350123 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.350085    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.354312 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.351950    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:12.354451 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.354421    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.355110 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.355088    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.356430 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.356407    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.357713 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.357696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:12.357812 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:12.357794    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:12.359158 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.359141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:12.360515 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.360493    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:12.361855 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.361837    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.363145 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.363124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:12.363243 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:12.363206    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:12.364537 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.364513    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:12.364642 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:12.364573    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:12.365710 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.365688    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:12.370750 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.370401    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4xzk4\""
Oct 30 00:04:12.371387 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.371366    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:04:12.371387 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.371370    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:04:12.371534 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.371414    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:04:12.371534 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.371484    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:04:12.374153 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.374132    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:04:12.374246 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.374190    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:04:12.374350 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.374336    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:04:12.398843 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.398827    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:04:12.402656 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.402636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-system-cni-dir\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.402747 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.402662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.402747 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.402679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52b7g\" (UniqueName: \"kubernetes.io/projected/23960d87-bf90-43c3-9ee6-464afa4c42ea-kube-api-access-52b7g\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.402747 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.402701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovn-node-metrics-cert\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.402905 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.402748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.402905 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.402811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-hostroot\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.402905 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.402866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/854339cf-9de2-4d3f-9275-20cb99cb7c89-host-slash\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:12.402905 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.402904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-ovn\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.403116 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.402922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-etc-selinux\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.403116 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.402947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wtc6\" (UniqueName: \"kubernetes.io/projected/8d2ad09d-f117-46ba-bf91-9dbcccca5748-kube-api-access-2wtc6\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.403116 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.402988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-netns\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.403116 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-kubelet\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.403116 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv7zp\" (UniqueName: \"kubernetes.io/projected/854339cf-9de2-4d3f-9275-20cb99cb7c89-kube-api-access-jv7zp\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:12.403323 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysconfig\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.403323 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysctl-d\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.403323 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysctl-conf\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.403323 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-host\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.403323 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-cnibin\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.403560 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.403560 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:12.403560 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-multus-certs\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.403560 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-etc-kubernetes\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.403560 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-run-netns\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.403560 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f9bp\" (UniqueName: \"kubernetes.io/projected/55ce16b2-ae85-4046-83b6-0fd872bca1a8-kube-api-access-4f9bp\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:12.403862 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-daemon-config\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.403862 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.403862 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-serviceca\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:12.403862 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-registration-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.403862 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403697    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-socket-dir-parent\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.403862 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-run\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.403862 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-sys\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.403862 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-tuned\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.403862 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-cni-bin\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.404202 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-k8s-cni-cncf-io\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.404202 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.403914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rffhf\" (UniqueName: \"kubernetes.io/projected/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-kube-api-access-rffhf\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.404202 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-kubernetes\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.404202 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-systemd\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.404202 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.404202 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-systemd\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.404522 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bf392012-ba7c-4999-b0fe-f0ae7cee60f7-konnectivity-ca\") pod \"konnectivity-agent-kmg9c\" (UID: \"bf392012-ba7c-4999-b0fe-f0ae7cee60f7\") " pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:12.404522 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-cni-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.404522 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404290    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-env-overrides\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.404522 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-host\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:12.404522 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-slash\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.404522 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-lib-modules\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.404522 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-etc-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.404522 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-socket-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.404522 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-system-cni-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.404522 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-cnibin\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpklh\" (UniqueName: \"kubernetes.io/projected/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-kube-api-access-gpklh\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xwzm\" (UniqueName: \"kubernetes.io/projected/d74677bd-f7f3-4a55-b5e7-0216819f5400-kube-api-access-7xwzm\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdwl\" (UniqueName: \"kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl\") pod \"network-check-target-tr4w8\" (UID: \"a055af25-0d4a-4b69-b395-259f12a4f363\") " pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw6bf\" (UniqueName: \"kubernetes.io/projected/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-kube-api-access-hw6bf\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-var-lib-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovnkube-config\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-cni-binary-copy\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-cni-bin\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-conf-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404899    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovnkube-script-lib\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.404981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-kubelet\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-run-ovn-kubernetes\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-cni-netd\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-var-lib-kubelet\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.405146 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-tmp\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e90ea1-6a7c-4007-89f4-bba585722b8a-tmp-dir\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-dbus\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-systemd-units\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405264    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-log-socket\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-sys-fs\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-cni-multus\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41e90ea1-6a7c-4007-89f4-bba585722b8a-hosts-file\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-node-log\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-modprobe-d\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mtkj\" (UniqueName: \"kubernetes.io/projected/41e90ea1-6a7c-4007-89f4-bba585722b8a-kube-api-access-8mtkj\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-kubelet-config\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-device-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bf392012-ba7c-4999-b0fe-f0ae7cee60f7-agent-certs\") pod \"konnectivity-agent-kmg9c\" (UID: \"bf392012-ba7c-4999-b0fe-f0ae7cee60f7\") " pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:12.405671 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.405653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/854339cf-9de2-4d3f-9275-20cb99cb7c89-iptables-alerter-script\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:12.434920 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:12.434892    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689ea6de40d51a70fe308b471157700c.slice/crio-f72d6d823f592be1cab0e2a8af30b7bf158bc2a39b11875d74a6a44a0dbb1ae6 WatchSource:0}: Error finding container f72d6d823f592be1cab0e2a8af30b7bf158bc2a39b11875d74a6a44a0dbb1ae6: Status 404 returned error can't find the container with id f72d6d823f592be1cab0e2a8af30b7bf158bc2a39b11875d74a6a44a0dbb1ae6
Oct 30 00:04:12.439921 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.439905    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:04:12.462149 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:12.462128    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3314e96d9e96380a527867d39d2fde5.slice/crio-59edc93c744b47b3b583bf3add9433bad217423f019c2e57e96c6a8c467214af WatchSource:0}: Error finding container 59edc93c744b47b3b583bf3add9433bad217423f019c2e57e96c6a8c467214af: Status 404 returned error can't find the container with id 59edc93c744b47b3b583bf3add9433bad217423f019c2e57e96c6a8c467214af
Oct 30 00:04:12.472780 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.472761    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:04:12.472858 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.472815    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mck7r"
Oct 30 00:04:12.479085 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.479061    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:04:12.479201 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.479182    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n7fsr\""
Oct 30 00:04:12.479201 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.479197    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:04:12.479315 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.479284    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fmnxz\""
Oct 30 00:04:12.479315 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.479300    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h92x8\""
Oct 30 00:04:12.479404 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.479306    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k8n8m\""
Oct 30 00:04:12.479404 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.479331    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wvd28\""
Oct 30 00:04:12.479404 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.479345    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r8lkn\""
Oct 30 00:04:12.479404 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.479348    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kct9w\""
Oct 30 00:04:12.479404 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.479370    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n6hlg\""
Oct 30 00:04:12.479904 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.479888    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:04:12.480026 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.480012    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:04:12.480076 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.480055    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:04:12.480076 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.480070    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:04:12.480185 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.480132    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:04:12.480185 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.480134    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:04:12.480185 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.480156    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:04:12.480185 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.480149    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:04:12.480185 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.480177    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:04:12.480336 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.480180    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:04:12.480336 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.480189    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:04:12.484548 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.484533    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:04:12.484609 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.484551    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:04:12.484609 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.484578    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:04:12.484609 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.484590    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:04:12.485588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.485568    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:04:12.485588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.485579    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:04:12.485588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.485587    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:04:12.506973 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.506923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-ovn\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.506973 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.506949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-etc-selinux\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.506973 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.506965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wtc6\" (UniqueName: \"kubernetes.io/projected/8d2ad09d-f117-46ba-bf91-9dbcccca5748-kube-api-access-2wtc6\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.507192 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.506983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-netns\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.507192 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.506999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-kubelet\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.507192 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7zp\" (UniqueName: \"kubernetes.io/projected/854339cf-9de2-4d3f-9275-20cb99cb7c89-kube-api-access-jv7zp\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:12.507192 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysconfig\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.507192 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507066    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-ovn\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.507192 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507083    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-netns\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.507192 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysctl-d\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.507192 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507111    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-etc-selinux\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.507192 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507105    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-kubelet\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.507192 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysctl-conf\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.507192 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507150    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysconfig\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.507192 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-host\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507209    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysctl-d\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-cnibin\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507247    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysctl-conf\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507239    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-host\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507251    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-cnibin\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-multus-certs\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-etc-kubernetes\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-run-netns\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507386    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-multus-certs\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507416    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-etc-kubernetes\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f9bp\" (UniqueName: \"kubernetes.io/projected/55ce16b2-ae85-4046-83b6-0fd872bca1a8-kube-api-access-4f9bp\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507438    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-run-netns\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-daemon-config\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:12.507407    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507510    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.507621 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:12.507553    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs podName:55ce16b2-ae85-4046-83b6-0fd872bca1a8 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:13.007531707 +0000 UTC m=+2.031231766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs") pod "network-metrics-daemon-6cnnk" (UID: "55ce16b2-ae85-4046-83b6-0fd872bca1a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-serviceca\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-registration-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-socket-dir-parent\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-run\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-sys\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507707    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-registration-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-tuned\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-cni-bin\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-k8s-cni-cncf-io\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507797    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-run\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507798    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-socket-dir-parent\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507812    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-sys\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507840    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-k8s-cni-cncf-io\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507845    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-cni-bin\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rffhf\" (UniqueName: \"kubernetes.io/projected/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-kube-api-access-rffhf\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.508345 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-kubernetes\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-systemd\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-systemd\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507961    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-systemd\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507963    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-systemd\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bf392012-ba7c-4999-b0fe-f0ae7cee60f7-konnectivity-ca\") pod \"konnectivity-agent-kmg9c\" (UID: \"bf392012-ba7c-4999-b0fe-f0ae7cee60f7\") " pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.507998    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-kubernetes\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-cni-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-env-overrides\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-host\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-slash\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-lib-modules\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508162    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-cni-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-etc-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-socket-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508195    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-etc-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-system-cni-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.509184 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508231    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-host\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-cnibin\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpklh\" (UniqueName: \"kubernetes.io/projected/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-kube-api-access-gpklh\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xwzm\" (UniqueName: \"kubernetes.io/projected/d74677bd-f7f3-4a55-b5e7-0216819f5400-kube-api-access-7xwzm\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508300    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-system-cni-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508302    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-lib-modules\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdwl\" (UniqueName: \"kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl\") pod \"network-check-target-tr4w8\" (UID: \"a055af25-0d4a-4b69-b395-259f12a4f363\") " pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw6bf\" (UniqueName: \"kubernetes.io/projected/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-kube-api-access-hw6bf\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-var-lib-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovnkube-config\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-cni-binary-copy\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-cni-bin\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-conf-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508252    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-slash\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovnkube-script-lib\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-kubelet\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.509920 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-run-ovn-kubernetes\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508591    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-run-ovn-kubernetes\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508642    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-cnibin\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508699    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-conf-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508703    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-daemon-config\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508422    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508754    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-socket-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-cni-netd\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508775    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-serviceca\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508797    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-var-lib-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508794    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-cni-bin\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508827    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-kubelet\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508847    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-cni-netd\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-var-lib-kubelet\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:12.508918    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508933    2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-tmp\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e90ea1-6a7c-4007-89f4-bba585722b8a-tmp-dir\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:12.510689 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.508989    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-var-lib-kubelet\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:12.509065    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret podName:3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:13.009047323 +0000 UTC m=+2.032747363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret") pod "global-pull-secret-syncer-v5tjs" (UID: "3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-dbus\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509114    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-cni-binary-copy\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-systemd-units\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509172    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-systemd-units\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509184    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-dbus\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-log-socket\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509223    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-log-socket\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-sys-fs\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-cni-multus\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509289    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e90ea1-6a7c-4007-89f4-bba585722b8a-tmp-dir\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509295    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41e90ea1-6a7c-4007-89f4-bba585722b8a-hosts-file\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-node-log\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509341    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-cni-multus\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.511642 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509350    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-sys-fs\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-modprobe-d\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtkj\" (UniqueName: \"kubernetes.io/projected/41e90ea1-6a7c-4007-89f4-bba585722b8a-kube-api-access-8mtkj\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509410    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41e90ea1-6a7c-4007-89f4-bba585722b8a-hosts-file\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-kubelet-config\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509468    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-modprobe-d\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509505    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-node-log\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-device-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509555    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-device-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509586    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bf392012-ba7c-4999-b0fe-f0ae7cee60f7-agent-certs\") pod \"konnectivity-agent-kmg9c\" (UID: \"bf392012-ba7c-4999-b0fe-f0ae7cee60f7\") " pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509585    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509627    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-kubelet-config\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/854339cf-9de2-4d3f-9275-20cb99cb7c89-iptables-alerter-script\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-system-cni-dir\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.512523 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52b7g\" (UniqueName: \"kubernetes.io/projected/23960d87-bf90-43c3-9ee6-464afa4c42ea-kube-api-access-52b7g\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509757    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-system-cni-dir\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.509970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovn-node-metrics-cert\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.510002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.510031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-hostroot\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.510069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/854339cf-9de2-4d3f-9275-20cb99cb7c89-host-slash\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.510159    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/854339cf-9de2-4d3f-9275-20cb99cb7c89-host-slash\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.510183    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bf392012-ba7c-4999-b0fe-f0ae7cee60f7-konnectivity-ca\") pod \"konnectivity-agent-kmg9c\" (UID: \"bf392012-ba7c-4999-b0fe-f0ae7cee60f7\") " pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.510251    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.510276    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.510294    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/854339cf-9de2-4d3f-9275-20cb99cb7c89-iptables-alerter-script\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.510299    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-hostroot\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.510687    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-env-overrides\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.510861    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovnkube-config\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.510912    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovnkube-script-lib\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.512883    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-tuned\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.512884    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovn-node-metrics-cert\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.512939    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-tmp\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.513194 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.513053    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bf392012-ba7c-4999-b0fe-f0ae7cee60f7-agent-certs\") pod \"konnectivity-agent-kmg9c\" (UID: \"bf392012-ba7c-4999-b0fe-f0ae7cee60f7\") " pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:12.534922 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.534775    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7zp\" (UniqueName: \"kubernetes.io/projected/854339cf-9de2-4d3f-9275-20cb99cb7c89-kube-api-access-jv7zp\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:12.534922 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:12.534919    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:04:12.535085 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.534926    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rffhf\" (UniqueName: \"kubernetes.io/projected/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-kube-api-access-rffhf\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.535085 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:12.534939    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:04:12.535085 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:12.534981    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ltdwl for pod openshift-network-diagnostics/network-check-target-tr4w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:04:12.535085 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:12.535050    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl podName:a055af25-0d4a-4b69-b395-259f12a4f363 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:13.035032068 +0000 UTC m=+2.058732143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ltdwl" (UniqueName: "kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl") pod "network-check-target-tr4w8" (UID: "a055af25-0d4a-4b69-b395-259f12a4f363") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:04:12.535085 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.535054    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f9bp\" (UniqueName: \"kubernetes.io/projected/55ce16b2-ae85-4046-83b6-0fd872bca1a8-kube-api-access-4f9bp\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:12.535826 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.535804    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wtc6\" (UniqueName: \"kubernetes.io/projected/8d2ad09d-f117-46ba-bf91-9dbcccca5748-kube-api-access-2wtc6\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.536865 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.536848    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw6bf\" (UniqueName: \"kubernetes.io/projected/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-kube-api-access-hw6bf\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:12.537437 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.537407    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtkj\" (UniqueName: \"kubernetes.io/projected/41e90ea1-6a7c-4007-89f4-bba585722b8a-kube-api-access-8mtkj\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:12.537505 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.537462    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xwzm\" (UniqueName: \"kubernetes.io/projected/d74677bd-f7f3-4a55-b5e7-0216819f5400-kube-api-access-7xwzm\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.537825 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.537329    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpklh\" (UniqueName: \"kubernetes.io/projected/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-kube-api-access-gpklh\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.537896 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.537882    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52b7g\" (UniqueName: \"kubernetes.io/projected/23960d87-bf90-43c3-9ee6-464afa4c42ea-kube-api-access-52b7g\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.604867 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.604846    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:04:12.627815 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.627791    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:04:12.678856 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.678836    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:12.682843 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.682821    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:12.689134 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:12.689109    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74677bd_f7f3_4a55_b5e7_0216819f5400.slice/crio-0da529af890d1e92daecf5aa5bf2521476c5588c65bd93615e3a4ed7bbea87b7 WatchSource:0}: Error finding container 0da529af890d1e92daecf5aa5bf2521476c5588c65bd93615e3a4ed7bbea87b7: Status 404 returned error can't find the container with id 0da529af890d1e92daecf5aa5bf2521476c5588c65bd93615e3a4ed7bbea87b7
Oct 30 00:04:12.692266 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:12.692239    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854339cf_9de2_4d3f_9275_20cb99cb7c89.slice/crio-3035c3dee7124a74822917e8cb6ae4859c1b34e56a6e7d1ac72eed640c4881ad WatchSource:0}: Error finding container 3035c3dee7124a74822917e8cb6ae4859c1b34e56a6e7d1ac72eed640c4881ad: Status 404 returned error can't find the container with id 3035c3dee7124a74822917e8cb6ae4859c1b34e56a6e7d1ac72eed640c4881ad
Oct 30 00:04:12.699183 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.699165    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:12.703472 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.703457    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:12.707059 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:12.707023    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2ad09d_f117_46ba_bf91_9dbcccca5748.slice/crio-b78005a723fff44b123d2bb9e6344b7175cd654ce789fcced15b22370bfbb7c6 WatchSource:0}: Error finding container b78005a723fff44b123d2bb9e6344b7175cd654ce789fcced15b22370bfbb7c6: Status 404 returned error can't find the container with id b78005a723fff44b123d2bb9e6344b7175cd654ce789fcced15b22370bfbb7c6
Oct 30 00:04:12.711913 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:12.711892    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccef4fa8_30d6_4e12_b2f1_470b13955cc5.slice/crio-078882be9302494ab45602cdffd919e440f03fea8f1fa23d5a642d1f78218f7a WatchSource:0}: Error finding container 078882be9302494ab45602cdffd919e440f03fea8f1fa23d5a642d1f78218f7a: Status 404 returned error can't find the container with id 078882be9302494ab45602cdffd919e440f03fea8f1fa23d5a642d1f78218f7a
Oct 30 00:04:12.725360 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.725339    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:12.734689 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:12.734669    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23960d87_bf90_43c3_9ee6_464afa4c42ea.slice/crio-73d77211ae7474d78bf3f1de0e8300480f343b294d4b0c6e684126fb3dc0ebf6 WatchSource:0}: Error finding container 73d77211ae7474d78bf3f1de0e8300480f343b294d4b0c6e684126fb3dc0ebf6: Status 404 returned error can't find the container with id 73d77211ae7474d78bf3f1de0e8300480f343b294d4b0c6e684126fb3dc0ebf6
Oct 30 00:04:12.740982 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.740966    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:12.746137 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.746118    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:12.748069 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:12.748053    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf392012_ba7c_4999_b0fe_f0ae7cee60f7.slice/crio-0f3dc17f2583b259fa05ecb1692f18956c129b0b91ff07ed5bdabe60dff5c0c0 WatchSource:0}: Error finding container 0f3dc17f2583b259fa05ecb1692f18956c129b0b91ff07ed5bdabe60dff5c0c0: Status 404 returned error can't find the container with id 0f3dc17f2583b259fa05ecb1692f18956c129b0b91ff07ed5bdabe60dff5c0c0
Oct 30 00:04:12.753804 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:12.753783    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e90ea1_6a7c_4007_89f4_bba585722b8a.slice/crio-d417a4f131c626416459d24c725bf4c0cc4c42aa1bfa316cd4e02c6c6f2ecddb WatchSource:0}: Error finding container d417a4f131c626416459d24c725bf4c0cc4c42aa1bfa316cd4e02c6c6f2ecddb: Status 404 returned error can't find the container with id d417a4f131c626416459d24c725bf4c0cc4c42aa1bfa316cd4e02c6c6f2ecddb
Oct 30 00:04:12.754294 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.754271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rrphh"
Oct 30 00:04:12.759755 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:12.759727    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:12.761927 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:12.761906    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e052a3c_35d6_4e5a_bd1c_8fd962f8effe.slice/crio-9ab4c5cff26f0ff20dffb1a927899c05e18dc4adbe4c66ea359d9010845a0e7f WatchSource:0}: Error finding container 9ab4c5cff26f0ff20dffb1a927899c05e18dc4adbe4c66ea359d9010845a0e7f: Status 404 returned error can't find the container with id 9ab4c5cff26f0ff20dffb1a927899c05e18dc4adbe4c66ea359d9010845a0e7f
Oct 30 00:04:12.767002 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:12.766971    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bc2c43_b4d6_47f7_86d3_5db7cdcf7f3f.slice/crio-241ece0fb72045fadb95614504bdf396f0149afbb4f0e9b3b8913f3bc43c83f7 WatchSource:0}: Error finding container 241ece0fb72045fadb95614504bdf396f0149afbb4f0e9b3b8913f3bc43c83f7: Status 404 returned error can't find the container with id 241ece0fb72045fadb95614504bdf396f0149afbb4f0e9b3b8913f3bc43c83f7
Oct 30 00:04:13.014095 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.014009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:13.014240 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.014102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:13.014240 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:13.014231    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:04:13.014364 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:13.014292    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret podName:3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:14.014273455 +0000 UTC m=+3.037973506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret") pod "global-pull-secret-syncer-v5tjs" (UID: "3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:04:13.014680 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:13.014663    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:04:13.014770 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:13.014716    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs podName:55ce16b2-ae85-4046-83b6-0fd872bca1a8 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:14.014699586 +0000 UTC m=+3.038399641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs") pod "network-metrics-daemon-6cnnk" (UID: "55ce16b2-ae85-4046-83b6-0fd872bca1a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:04:13.114895 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.114858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdwl\" (UniqueName: \"kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl\") pod \"network-check-target-tr4w8\" (UID: \"a055af25-0d4a-4b69-b395-259f12a4f363\") " pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:13.115088 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:13.115002    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:04:13.115088 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:13.115017    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:04:13.115088 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:13.115026    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ltdwl for pod openshift-network-diagnostics/network-check-target-tr4w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:04:13.115240 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:13.115090    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl podName:a055af25-0d4a-4b69-b395-259f12a4f363 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:14.115072041 +0000 UTC m=+3.138772091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ltdwl" (UniqueName: "kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl") pod "network-check-target-tr4w8" (UID: "a055af25-0d4a-4b69-b395-259f12a4f363") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:04:13.457172 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.456991    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal" event={"ID":"689ea6de40d51a70fe308b471157700c","Type":"ContainerStarted","Data":"f72d6d823f592be1cab0e2a8af30b7bf158bc2a39b11875d74a6a44a0dbb1ae6"}
Oct 30 00:04:13.462532 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.462500    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6hq6g" event={"ID":"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f","Type":"ContainerStarted","Data":"241ece0fb72045fadb95614504bdf396f0149afbb4f0e9b3b8913f3bc43c83f7"}
Oct 30 00:04:13.465632 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.465603    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kmg9c" event={"ID":"bf392012-ba7c-4999-b0fe-f0ae7cee60f7","Type":"ContainerStarted","Data":"0f3dc17f2583b259fa05ecb1692f18956c129b0b91ff07ed5bdabe60dff5c0c0"}
Oct 30 00:04:13.470663 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.470637    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wtbxs" event={"ID":"ccef4fa8-30d6-4e12-b2f1-470b13955cc5","Type":"ContainerStarted","Data":"078882be9302494ab45602cdffd919e440f03fea8f1fa23d5a642d1f78218f7a"}
Oct 30 00:04:13.474912 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.474856    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:59:12 +0000 UTC" deadline="2028-05-01 07:43:11.036651376 +0000 UTC"
Oct 30 00:04:13.475001 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.474913    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13183h38m57.56174204s"
Oct 30 00:04:13.479231 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.479136    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k" event={"ID":"8d2ad09d-f117-46ba-bf91-9dbcccca5748","Type":"ContainerStarted","Data":"b78005a723fff44b123d2bb9e6344b7175cd654ce789fcced15b22370bfbb7c6"}
Oct 30 00:04:13.482090 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.482007    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rrphh" event={"ID":"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe","Type":"ContainerStarted","Data":"9ab4c5cff26f0ff20dffb1a927899c05e18dc4adbe4c66ea359d9010845a0e7f"}
Oct 30 00:04:13.492784 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.492755    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jvrpq" event={"ID":"41e90ea1-6a7c-4007-89f4-bba585722b8a","Type":"ContainerStarted","Data":"d417a4f131c626416459d24c725bf4c0cc4c42aa1bfa316cd4e02c6c6f2ecddb"}
Oct 30 00:04:13.512411 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.512383    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8k6ms" event={"ID":"23960d87-bf90-43c3-9ee6-464afa4c42ea","Type":"ContainerStarted","Data":"73d77211ae7474d78bf3f1de0e8300480f343b294d4b0c6e684126fb3dc0ebf6"}
Oct 30 00:04:13.525509 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.525482    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hnp8z" event={"ID":"854339cf-9de2-4d3f-9275-20cb99cb7c89","Type":"ContainerStarted","Data":"3035c3dee7124a74822917e8cb6ae4859c1b34e56a6e7d1ac72eed640c4881ad"}
Oct 30 00:04:13.549738 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.549712    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz" event={"ID":"d74677bd-f7f3-4a55-b5e7-0216819f5400","Type":"ContainerStarted","Data":"0da529af890d1e92daecf5aa5bf2521476c5588c65bd93615e3a4ed7bbea87b7"}
Oct 30 00:04:13.554444 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.554418    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal" event={"ID":"e3314e96d9e96380a527867d39d2fde5","Type":"ContainerStarted","Data":"59edc93c744b47b3b583bf3add9433bad217423f019c2e57e96c6a8c467214af"}
Oct 30 00:04:13.842349 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:13.842271    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:04:14.022590 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:14.021811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:14.022590 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:14.021889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:14.022590 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:14.022022    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:04:14.022590 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:14.022099    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret podName:3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:16.022081497 +0000 UTC m=+5.045781547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret") pod "global-pull-secret-syncer-v5tjs" (UID: "3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:04:14.022590 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:14.022500    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:04:14.022590 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:14.022554    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs podName:55ce16b2-ae85-4046-83b6-0fd872bca1a8 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:16.022538234 +0000 UTC m=+5.046238285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs") pod "network-metrics-daemon-6cnnk" (UID: "55ce16b2-ae85-4046-83b6-0fd872bca1a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:04:14.122576 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:14.122504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdwl\" (UniqueName: \"kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl\") pod \"network-check-target-tr4w8\" (UID: \"a055af25-0d4a-4b69-b395-259f12a4f363\") " pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:14.122744 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:14.122726    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:04:14.122799 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:14.122757    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:04:14.122799 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:14.122772    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ltdwl for pod openshift-network-diagnostics/network-check-target-tr4w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:04:14.122865 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:14.122829    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl podName:a055af25-0d4a-4b69-b395-259f12a4f363 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:16.122810753 +0000 UTC m=+5.146510793 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ltdwl" (UniqueName: "kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl") pod "network-check-target-tr4w8" (UID: "a055af25-0d4a-4b69-b395-259f12a4f363") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:04:14.436016 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:14.435945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:14.436200 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:14.436096    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:14.437139 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:14.436598    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:14.437139 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:14.436717    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:14.437139 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:14.436781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:14.437139 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:14.436855    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:14.476057 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:14.475999    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:59:12 +0000 UTC" deadline="2028-06-01 16:50:12.549445939 +0000 UTC"
Oct 30 00:04:14.476057 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:14.476057    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13936h45m58.073394309s"
Oct 30 00:04:16.043057 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:16.043005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:16.043562 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:16.043110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:16.043562 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:16.043372    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:04:16.043562 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:16.043441    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs podName:55ce16b2-ae85-4046-83b6-0fd872bca1a8 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:20.043419739 +0000 UTC m=+9.067119780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs") pod "network-metrics-daemon-6cnnk" (UID: "55ce16b2-ae85-4046-83b6-0fd872bca1a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:04:16.044191 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:16.043867    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:04:16.044191 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:16.043922    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret podName:3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:20.043906616 +0000 UTC m=+9.067606655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret") pod "global-pull-secret-syncer-v5tjs" (UID: "3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:04:16.144231 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:16.143489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdwl\" (UniqueName: \"kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl\") pod \"network-check-target-tr4w8\" (UID: \"a055af25-0d4a-4b69-b395-259f12a4f363\") " pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:16.144231 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:16.143719    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:04:16.144231 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:16.143763    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:04:16.144231 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:16.143777    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ltdwl for pod openshift-network-diagnostics/network-check-target-tr4w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:04:16.144231 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:16.143839    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl podName:a055af25-0d4a-4b69-b395-259f12a4f363 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:20.143820472 +0000 UTC m=+9.167520523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ltdwl" (UniqueName: "kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl") pod "network-check-target-tr4w8" (UID: "a055af25-0d4a-4b69-b395-259f12a4f363") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:04:16.436867 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:16.436001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:16.436867 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:16.436152    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:16.436867 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:16.436572    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:16.436867 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:16.436689    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:16.436867 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:16.436759    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:16.436867 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:16.436832    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:17.632403 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:17.632086    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kmg9c" event={"ID":"bf392012-ba7c-4999-b0fe-f0ae7cee60f7","Type":"ContainerStarted","Data":"f0703cfae17a3265af48b0ad1289a5494ab190e4e825203b8c51f0b103c8e573"}
Oct 30 00:04:17.635279 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:17.635251    2581 generic.go:356] "Generic (PLEG): container finished" podID="23960d87-bf90-43c3-9ee6-464afa4c42ea" containerID="6803578ebd7b12ede7330bd358a899512d064e3c309417775a5139be0df8d95e" exitCode=0
Oct 30 00:04:17.635871 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:17.635851    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8k6ms" event={"ID":"23960d87-bf90-43c3-9ee6-464afa4c42ea","Type":"ContainerDied","Data":"6803578ebd7b12ede7330bd358a899512d064e3c309417775a5139be0df8d95e"}
Oct 30 00:04:17.640466 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:17.640445    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal" event={"ID":"e3314e96d9e96380a527867d39d2fde5","Type":"ContainerStarted","Data":"cac5ad08c9a57b7051ca6abf9de2ecb4ded9fd974b32663dcab1b079a65e3e4c"}
Oct 30 00:04:17.644368 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:17.643961    2581 generic.go:356] "Generic (PLEG): container finished" podID="689ea6de40d51a70fe308b471157700c" containerID="d17c2fb640d1cce2b5bcd574a50285189357d76a1834555438b7ba5040d0ced2" exitCode=0
Oct 30 00:04:17.644368 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:17.643996    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal" event={"ID":"689ea6de40d51a70fe308b471157700c","Type":"ContainerDied","Data":"d17c2fb640d1cce2b5bcd574a50285189357d76a1834555438b7ba5040d0ced2"}
Oct 30 00:04:17.672451 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:17.672343    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kmg9c" podStartSLOduration=2.291500841 podStartE2EDuration="6.672332819s" totalImagesPullingTime="4.380831978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:17.671975052 +0000 UTC m=+6.695675107" watchObservedRunningTime="2026-10-30 00:04:17.672332819 +0000 UTC m=+6.696032873"
Oct 30 00:04:18.435311 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:18.435205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:18.435490 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:18.435346    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:18.435761 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:18.435739    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:18.435858 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:18.435840    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:18.435915 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:18.435909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:18.436007 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:18.435987    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:18.653199 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:18.653166    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal" event={"ID":"689ea6de40d51a70fe308b471157700c","Type":"ContainerStarted","Data":"c2e2c091413db34aefbfbf9152575fc056a8c5b874932db9cda447e2253280d4"}
Oct 30 00:04:18.692392 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:18.692296    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal" podStartSLOduration=7.692277017 podStartE2EDuration="7.692277017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:17.794097867 +0000 UTC m=+6.817797924" watchObservedRunningTime="2026-10-30 00:04:18.692277017 +0000 UTC m=+7.715977073"
Oct 30 00:04:18.692626 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:18.692594    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal" podStartSLOduration=7.692586786 podStartE2EDuration="7.692586786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:18.691653818 +0000 UTC m=+7.715353874" watchObservedRunningTime="2026-10-30 00:04:18.692586786 +0000 UTC m=+7.716286840"
Oct 30 00:04:19.656570 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:19.656533    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6hq6g" event={"ID":"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f","Type":"ContainerStarted","Data":"edb305d613e7a75471bcdd17b0cae341d5cc3cf58a0222160502fe8aa953bebf"}
Oct 30 00:04:19.661518 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:19.661495    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k" event={"ID":"8d2ad09d-f117-46ba-bf91-9dbcccca5748","Type":"ContainerStarted","Data":"eb12a24482bcdfd91bc56e9df5f18cd42d718500c2b32d321e54a66c1ee05ed0"}
Oct 30 00:04:19.695995 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:19.695951    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6hq6g" podStartSLOduration=2.796802944 podStartE2EDuration="8.695938184s" totalImagesPullingTime="5.89913524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:19.695352979 +0000 UTC m=+8.719053036" watchObservedRunningTime="2026-10-30 00:04:19.695938184 +0000 UTC m=+8.719638238"
Oct 30 00:04:20.086889 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:20.086861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:20.087039 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:20.086923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:20.087108 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:20.087061    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:04:20.087171 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:20.087115    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs podName:55ce16b2-ae85-4046-83b6-0fd872bca1a8 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:28.087098396 +0000 UTC m=+17.110798436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs") pod "network-metrics-daemon-6cnnk" (UID: "55ce16b2-ae85-4046-83b6-0fd872bca1a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:04:20.087503 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:20.087484    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:04:20.087590 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:20.087536    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret podName:3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:28.087522762 +0000 UTC m=+17.111222813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret") pod "global-pull-secret-syncer-v5tjs" (UID: "3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:04:20.188104 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:20.188069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdwl\" (UniqueName: \"kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl\") pod \"network-check-target-tr4w8\" (UID: \"a055af25-0d4a-4b69-b395-259f12a4f363\") " pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:20.188258 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:20.188241    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:04:20.188258 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:20.188258    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:04:20.188371 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:20.188270    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ltdwl for pod openshift-network-diagnostics/network-check-target-tr4w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:04:20.188371 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:20.188320    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl podName:a055af25-0d4a-4b69-b395-259f12a4f363 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:28.188303898 +0000 UTC m=+17.212003949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ltdwl" (UniqueName: "kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl") pod "network-check-target-tr4w8" (UID: "a055af25-0d4a-4b69-b395-259f12a4f363") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:04:20.436408 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:20.435532    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:20.436408 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:20.435678    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:20.436408 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:20.436084    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:20.436408 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:20.436185    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:20.436408 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:20.436231    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:20.436408 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:20.436317    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:20.838644 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:20.838242    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:20.840715 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:20.840643    2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:04:21.169087 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:21.167992    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:21.169087 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:21.168811    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:21.353048 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:21.352718    2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:04:20.840659965Z","UUID":"bc863500-4ca6-4dc6-b2b9-dd535f416578","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:04:21.356524 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:21.356410    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:04:21.356524 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:21.356439    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:04:21.668966 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:21.668938    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k" event={"ID":"8d2ad09d-f117-46ba-bf91-9dbcccca5748","Type":"ContainerStarted","Data":"80399205c9985d835b5a65e70427720d830145a623ef24f35ed3c76d4282120d"}
Oct 30 00:04:21.671298 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:21.671267    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rrphh" event={"ID":"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe","Type":"ContainerStarted","Data":"fe075215ba640334dee99589e36a281c17d3024d7fb2b084654e649ecb5069be"}
Oct 30 00:04:21.675130 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:21.674800    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jvrpq" event={"ID":"41e90ea1-6a7c-4007-89f4-bba585722b8a","Type":"ContainerStarted","Data":"043a945152dc469b24428bc15c6f95cee10a41f9b91ee6e65285aa5eceda960d"}
Oct 30 00:04:21.675450 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:21.675408    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:21.718432 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:21.718384    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rrphh" podStartSLOduration=2.6471804370000003 podStartE2EDuration="10.718368727s" totalImagesPullingTime="8.07118829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:21.717630198 +0000 UTC m=+10.741330254" watchObservedRunningTime="2026-10-30 00:04:21.718368727 +0000 UTC m=+10.742068781"
Oct 30 00:04:21.800271 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:21.800208    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jvrpq" podStartSLOduration=2.370293597 podStartE2EDuration="10.800187957s" totalImagesPullingTime="8.42989436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:21.799637621 +0000 UTC m=+10.823337677" watchObservedRunningTime="2026-10-30 00:04:21.800187957 +0000 UTC m=+10.823888013"
Oct 30 00:04:22.436353 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:22.436160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:22.436779 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:22.436176    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:22.436779 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:22.436224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:22.436779 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:22.436457    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:22.436779 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:22.436546    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:22.436779 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:22.436622    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:22.688091 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:22.687778    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k" event={"ID":"8d2ad09d-f117-46ba-bf91-9dbcccca5748","Type":"ContainerStarted","Data":"7c8eb11ee7473c221ee18ea7a8bb6b869eb034f456c068085d31c79b8ef1980d"}
Oct 30 00:04:22.689879 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:22.689851    2581 generic.go:356] "Generic (PLEG): container finished" podID="23960d87-bf90-43c3-9ee6-464afa4c42ea" containerID="8aedfc13ee033075736ecf300dc56ccb41340847e4f16a20256f384a5f972eac" exitCode=0
Oct 30 00:04:22.690013 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:22.689944    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8k6ms" event={"ID":"23960d87-bf90-43c3-9ee6-464afa4c42ea","Type":"ContainerDied","Data":"8aedfc13ee033075736ecf300dc56ccb41340847e4f16a20256f384a5f972eac"}
Oct 30 00:04:22.732772 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:22.732728    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k" podStartSLOduration=2.930754939 podStartE2EDuration="11.732715963s" totalImagesPullingTime="8.801961024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:22.732386998 +0000 UTC m=+11.756087054" watchObservedRunningTime="2026-10-30 00:04:22.732715963 +0000 UTC m=+11.756416018"
Oct 30 00:04:23.694686 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:23.694652    2581 generic.go:356] "Generic (PLEG): container finished" podID="23960d87-bf90-43c3-9ee6-464afa4c42ea" containerID="971c3512a44a0f0619254b977f8bd02384582868883a5160994c398bed82370a" exitCode=0
Oct 30 00:04:23.695245 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:23.694696    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8k6ms" event={"ID":"23960d87-bf90-43c3-9ee6-464afa4c42ea","Type":"ContainerDied","Data":"971c3512a44a0f0619254b977f8bd02384582868883a5160994c398bed82370a"}
Oct 30 00:04:23.696401 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:23.696376    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hnp8z" event={"ID":"854339cf-9de2-4d3f-9275-20cb99cb7c89","Type":"ContainerStarted","Data":"e18a97a5d1ea44e5af41de516ef4253c67fec9143d98c2bbf1501824827382d0"}
Oct 30 00:04:23.698172 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:23.698149    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wtbxs" event={"ID":"ccef4fa8-30d6-4e12-b2f1-470b13955cc5","Type":"ContainerStarted","Data":"7240f729bf65e292ff614e66aed2297aaf4bddda4320085c59432455c66cff6e"}
Oct 30 00:04:23.790516 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:23.790467    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hnp8z" podStartSLOduration=4.150945057 podStartE2EDuration="12.790452294s" totalImagesPullingTime="8.639507237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:23.790167675 +0000 UTC m=+12.813867731" watchObservedRunningTime="2026-10-30 00:04:23.790452294 +0000 UTC m=+12.814152352"
Oct 30 00:04:23.831662 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:23.831616    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wtbxs" podStartSLOduration=2.377336056 podStartE2EDuration="12.83160361s" totalImagesPullingTime="10.454267554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:23.831057659 +0000 UTC m=+12.854757711" watchObservedRunningTime="2026-10-30 00:04:23.83160361 +0000 UTC m=+12.855303664"
Oct 30 00:04:24.435360 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:24.435333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:24.435538 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:24.435392    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:24.435538 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:24.435434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:24.435648 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:24.435535    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:24.435648 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:24.435611    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:24.435724 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:24.435693    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:25.704785 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:25.704751    2581 generic.go:356] "Generic (PLEG): container finished" podID="23960d87-bf90-43c3-9ee6-464afa4c42ea" containerID="3ecf0d5e69bc467e97200bc2687c98d902f9d3e03f42f1cf82ce142fc816051a" exitCode=0
Oct 30 00:04:25.705526 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:25.704795    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8k6ms" event={"ID":"23960d87-bf90-43c3-9ee6-464afa4c42ea","Type":"ContainerDied","Data":"3ecf0d5e69bc467e97200bc2687c98d902f9d3e03f42f1cf82ce142fc816051a"}
Oct 30 00:04:26.435747 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:26.435499    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:26.435901 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:26.435866    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:26.435901 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:26.435592    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:26.435993 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:26.435955    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:26.435993 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:26.435530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:26.436587 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:26.436037    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:28.150893 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.150867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:28.151192 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.150919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:28.151192 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:28.151012    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:04:28.151192 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:28.151082    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret podName:3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:44.151055561 +0000 UTC m=+33.174755593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret") pod "global-pull-secret-syncer-v5tjs" (UID: "3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:04:28.151192 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:28.151012    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:04:28.151192 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:28.151110    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs podName:55ce16b2-ae85-4046-83b6-0fd872bca1a8 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:44.151104622 +0000 UTC m=+33.174804655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs") pod "network-metrics-daemon-6cnnk" (UID: "55ce16b2-ae85-4046-83b6-0fd872bca1a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:04:28.251580 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.251527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdwl\" (UniqueName: \"kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl\") pod \"network-check-target-tr4w8\" (UID: \"a055af25-0d4a-4b69-b395-259f12a4f363\") " pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:28.251673 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:28.251640    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:04:28.251673 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:28.251652    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:04:28.251673 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:28.251660    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ltdwl for pod openshift-network-diagnostics/network-check-target-tr4w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:04:28.251759 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:28.251697    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl podName:a055af25-0d4a-4b69-b395-259f12a4f363 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:44.251686188 +0000 UTC m=+33.275386222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ltdwl" (UniqueName: "kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl") pod "network-check-target-tr4w8" (UID: "a055af25-0d4a-4b69-b395-259f12a4f363") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:04:28.435455 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.435434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:28.435571 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.435549    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:28.435571 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:28.435553    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:28.435801 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:28.435626    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:28.435801 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.435648    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:28.435801 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:28.435700    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:28.712713 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.712658    2581 generic.go:356] "Generic (PLEG): container finished" podID="23960d87-bf90-43c3-9ee6-464afa4c42ea" containerID="2d00de2dc928ce79d59da7729c2a6896810a59639585ed7eee0b7701981d6628" exitCode=0
Oct 30 00:04:28.712816 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.712745    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8k6ms" event={"ID":"23960d87-bf90-43c3-9ee6-464afa4c42ea","Type":"ContainerDied","Data":"2d00de2dc928ce79d59da7729c2a6896810a59639585ed7eee0b7701981d6628"}
Oct 30 00:04:28.715570 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.715554    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:04:28.715950 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.715934    2581 generic.go:356] "Generic (PLEG): container finished" podID="d74677bd-f7f3-4a55-b5e7-0216819f5400" containerID="742b573dce28a53b4c5075bb2ab55d8e9b84860536ed704eabf75eee73dc5b36" exitCode=1
Oct 30 00:04:28.716002 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.715963    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz" event={"ID":"d74677bd-f7f3-4a55-b5e7-0216819f5400","Type":"ContainerStarted","Data":"cb81be875de9b83311d53faea8942d74bc9b6b51588e4787c4038e01d76d8258"}
Oct 30 00:04:28.716047 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.716012    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz" event={"ID":"d74677bd-f7f3-4a55-b5e7-0216819f5400","Type":"ContainerStarted","Data":"686037801e231baee42e38f5c5d071efa410e831934c616c5135daea13fdab06"}
Oct 30 00:04:28.716087 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.716066    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz" event={"ID":"d74677bd-f7f3-4a55-b5e7-0216819f5400","Type":"ContainerStarted","Data":"a4935bfb3fe90e35caa456b605d9199cd90986494e8fbe777d17eb3b8b2728f1"}
Oct 30 00:04:28.716116 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.716098    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz" event={"ID":"d74677bd-f7f3-4a55-b5e7-0216819f5400","Type":"ContainerStarted","Data":"d5ef971c83a7bb94c96840b3d5ecfab5e1a38167d14193b19d036014c3958e17"}
Oct 30 00:04:28.716147 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.716124    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz" event={"ID":"d74677bd-f7f3-4a55-b5e7-0216819f5400","Type":"ContainerDied","Data":"742b573dce28a53b4c5075bb2ab55d8e9b84860536ed704eabf75eee73dc5b36"}
Oct 30 00:04:28.716174 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:28.716159    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz" event={"ID":"d74677bd-f7f3-4a55-b5e7-0216819f5400","Type":"ContainerStarted","Data":"9b082025e242ed35319c93d48552a46427847e7cbdc1cc23290c01f8e72ce0b9"}
Oct 30 00:04:29.720089 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:29.720058    2581 generic.go:356] "Generic (PLEG): container finished" podID="23960d87-bf90-43c3-9ee6-464afa4c42ea" containerID="5f2680238bd6cec7df87d7116835a1ce3b0660d58410d3e7612cccda256a677b" exitCode=0
Oct 30 00:04:29.720460 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:29.720119    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8k6ms" event={"ID":"23960d87-bf90-43c3-9ee6-464afa4c42ea","Type":"ContainerDied","Data":"5f2680238bd6cec7df87d7116835a1ce3b0660d58410d3e7612cccda256a677b"}
Oct 30 00:04:30.435970 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:30.435932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:30.436152 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:30.435951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:30.436152 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:30.436063    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:30.436152 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:30.436066    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:30.436152 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:30.436133    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:30.436275 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:30.436210    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:30.725681 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:30.725592    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8k6ms" event={"ID":"23960d87-bf90-43c3-9ee6-464afa4c42ea","Type":"ContainerStarted","Data":"421c17b4bab6043c16e91e5ce67d88933b3697fe4cc8363ed5c3fbdf755033bf"}
Oct 30 00:04:30.728874 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:30.728854    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:04:30.729201 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:30.729182    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz" event={"ID":"d74677bd-f7f3-4a55-b5e7-0216819f5400","Type":"ContainerStarted","Data":"ab0108b8c8f66c327b532f5d3998427cf2ca16b00f39d6acdc1b9f3be004f3d1"}
Oct 30 00:04:30.801442 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:30.801405    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8k6ms" podStartSLOduration=9.026095221 podStartE2EDuration="19.801395318s" totalImagesPullingTime="10.775300097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:30.787252358 +0000 UTC m=+19.810952407" watchObservedRunningTime="2026-10-30 00:04:30.801395318 +0000 UTC m=+19.825095372"
Oct 30 00:04:32.435588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:32.435559    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:32.435588 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:32.435590    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:32.436340 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:32.435667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:32.436340 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:32.435682    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:32.436340 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:32.435770    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:32.436340 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:32.435892    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:32.737081 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:32.736989    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:04:32.737416 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:32.737396    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz" event={"ID":"d74677bd-f7f3-4a55-b5e7-0216819f5400","Type":"ContainerStarted","Data":"663c62db7f9e5533483b4fa4d409e6dd6d40ba1592b463f9384190733b033dd9"}
Oct 30 00:04:32.737686 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:32.737632    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:32.737838 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:32.737822    2581 scope.go:122] "RemoveContainer" containerID="742b573dce28a53b4c5075bb2ab55d8e9b84860536ed704eabf75eee73dc5b36"
Oct 30 00:04:32.753124 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:32.753105    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:33.743178 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:33.743149    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:04:33.744182 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:33.744154    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz" event={"ID":"d74677bd-f7f3-4a55-b5e7-0216819f5400","Type":"ContainerStarted","Data":"038fcfb261af8052b7788944ce68b4aa390b7412aa7f1f295760e584aaf0d9b6"}
Oct 30 00:04:33.744730 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:33.744707    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:33.744910 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:33.744884    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:33.759244 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:33.759225    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:33.842971 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:33.842923    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz" podStartSLOduration=7.805251609 podStartE2EDuration="22.842909965s" totalImagesPullingTime="15.037658356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:33.842906361 +0000 UTC m=+22.866606416" watchObservedRunningTime="2026-10-30 00:04:33.842909965 +0000 UTC m=+22.866610020"
Oct 30 00:04:34.378409 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:34.378337    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v5tjs"]
Oct 30 00:04:34.379040 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:34.379000    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:34.379288 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:34.379262    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:34.382455 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:34.382430    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tr4w8"]
Oct 30 00:04:34.382615 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:34.382595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:34.382749 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:34.382727    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:34.383548 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:34.383524    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6cnnk"]
Oct 30 00:04:34.383646 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:34.383616    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:34.383710 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:34.383694    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:36.435211 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:36.435184    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:36.435578 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:36.435216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:36.435578 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:36.435288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:36.435578 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:36.435308    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:36.435578 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:36.435405    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:36.435578 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:36.435535    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:38.436186 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:38.436149    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:38.436659 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:38.436189    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:38.436659 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:38.436194    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:38.436659 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:38.436278    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:38.436659 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:38.436330    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:38.436659 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:38.436411    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:40.436237 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.436203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:40.436569 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.436272    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:40.436569 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.436300    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:40.436569 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:40.436397    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6cnnk" podUID="55ce16b2-ae85-4046-83b6-0fd872bca1a8"
Oct 30 00:04:40.436569 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:40.436453    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tr4w8" podUID="a055af25-0d4a-4b69-b395-259f12a4f363"
Oct 30 00:04:40.436569 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:40.436515    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5tjs" podUID="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5"
Oct 30 00:04:40.792276 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.792210    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.ec2.internal" event="NodeReady"
Oct 30 00:04:40.792449 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.792329    2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 00:04:40.852959 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.852932    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v"]
Oct 30 00:04:40.861546 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.861522    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qxjrt"]
Oct 30 00:04:40.861692 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.861647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v"
Oct 30 00:04:40.867687 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.867667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxjrt"
Oct 30 00:04:40.873670 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.873652    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v"]
Oct 30 00:04:40.877416 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.877393    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxjrt"]
Oct 30 00:04:40.880018 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.879997    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xvj9n"]
Oct 30 00:04:40.882430 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.882413    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-l5flb\""
Oct 30 00:04:40.882525 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.882467    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 00:04:40.882525 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.882494    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 00:04:40.884720 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.884704    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:04:40.887230 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.887215    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ntcgl\""
Oct 30 00:04:40.888901 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.888888    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:40.903260 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.903236    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xvj9n"]
Oct 30 00:04:40.946262 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.946237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxq4w\" (UniqueName: \"kubernetes.io/projected/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3-kube-api-access-xxq4w\") pod \"ingress-canary-qxjrt\" (UID: \"9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3\") " pod="openshift-ingress-canary/ingress-canary-qxjrt"
Oct 30 00:04:40.946373 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.946279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3-cert\") pod \"ingress-canary-qxjrt\" (UID: \"9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3\") " pod="openshift-ingress-canary/ingress-canary-qxjrt"
Oct 30 00:04:40.946373 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.946300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwjfl\" (UniqueName: \"kubernetes.io/projected/66a3d601-1733-40ac-834e-7378e3098542-kube-api-access-nwjfl\") pod \"migrator-66df5d496b-4jj6v\" (UID: \"66a3d601-1733-40ac-834e-7378e3098542\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v"
Oct 30 00:04:40.955962 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.955939    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xzfnp"]
Oct 30 00:04:40.965134 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.965111    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:40.977524 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.977503    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xzfnp"]
Oct 30 00:04:40.982563 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.982540    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:04:40.983196 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.983165    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:04:40.983282 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.983197    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:04:40.983282 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.983205    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:04:40.983282 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.983220    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqlwc\""
Oct 30 00:04:40.983282 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.983228    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:04:40.983282 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.983226    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:04:40.983282 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.983256    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:04:40.983282 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.983276    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9vr82\""
Oct 30 00:04:40.983519 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:40.983290    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:04:41.046758 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.046693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-data-volume\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.046758 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.046722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-crio-socket\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.046758 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.046739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:41.046984 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.046837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxq4w\" (UniqueName: \"kubernetes.io/projected/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3-kube-api-access-xxq4w\") pod \"ingress-canary-qxjrt\" (UID: \"9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3\") " pod="openshift-ingress-canary/ingress-canary-qxjrt"
Oct 30 00:04:41.046984 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.046878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpx4v\" (UniqueName: \"kubernetes.io/projected/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-kube-api-access-bpx4v\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:41.046984 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.046922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.046984 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.046973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-config-volume\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:41.047242 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.047020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-tmp-dir\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:41.047242 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.047107    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3-cert\") pod \"ingress-canary-qxjrt\" (UID: \"9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3\") " pod="openshift-ingress-canary/ingress-canary-qxjrt"
Oct 30 00:04:41.047242 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.047151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.047242 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.047182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pbbk\" (UniqueName: \"kubernetes.io/projected/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-api-access-8pbbk\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.047242 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.047214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwjfl\" (UniqueName: \"kubernetes.io/projected/66a3d601-1733-40ac-834e-7378e3098542-kube-api-access-nwjfl\") pod \"migrator-66df5d496b-4jj6v\" (UID: \"66a3d601-1733-40ac-834e-7378e3098542\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v"
Oct 30 00:04:41.051017 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.050982    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3-cert\") pod \"ingress-canary-qxjrt\" (UID: \"9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3\") " pod="openshift-ingress-canary/ingress-canary-qxjrt"
Oct 30 00:04:41.072059 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.072039    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxq4w\" (UniqueName: \"kubernetes.io/projected/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3-kube-api-access-xxq4w\") pod \"ingress-canary-qxjrt\" (UID: \"9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3\") " pod="openshift-ingress-canary/ingress-canary-qxjrt"
Oct 30 00:04:41.072245 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.072226    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwjfl\" (UniqueName: \"kubernetes.io/projected/66a3d601-1733-40ac-834e-7378e3098542-kube-api-access-nwjfl\") pod \"migrator-66df5d496b-4jj6v\" (UID: \"66a3d601-1733-40ac-834e-7378e3098542\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v"
Oct 30 00:04:41.148331 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.148306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-data-volume\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.148471 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.148337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-crio-socket\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.148471 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.148361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:41.148471 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.148393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpx4v\" (UniqueName: \"kubernetes.io/projected/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-kube-api-access-bpx4v\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:41.148471 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.148424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.148471 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.148441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-config-volume\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:41.148471 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.148457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-tmp-dir\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:41.148738 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:41.148472    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:04:41.148738 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.148493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.148738 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:41.148540    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls podName:3f2407f8-44bd-4a86-af6d-66429fb7ecfd nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.648521324 +0000 UTC m=+30.672221387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls") pod "dns-default-xzfnp" (UID: "3f2407f8-44bd-4a86-af6d-66429fb7ecfd") : secret "dns-default-metrics-tls" not found
Oct 30 00:04:41.148738 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.148571    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-crio-socket\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.148738 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.148699    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-data-volume\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.149010 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:41.148993    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:41.149073 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.149035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pbbk\" (UniqueName: \"kubernetes.io/projected/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-api-access-8pbbk\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.149108 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:41.149101    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls podName:657fcf3d-0fff-4894-94dc-2dc1b84a9faf nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.649090701 +0000 UTC m=+30.672790733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvj9n" (UID: "657fcf3d-0fff-4894-94dc-2dc1b84a9faf") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:41.149206 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:41.149190    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:04:41.149286 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:41.149267    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm podName:657fcf3d-0fff-4894-94dc-2dc1b84a9faf nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.649248768 +0000 UTC m=+30.672948830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvj9n" (UID: "657fcf3d-0fff-4894-94dc-2dc1b84a9faf") : configmap "kube-rbac-proxy" not found
Oct 30 00:04:41.149463 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.149448    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-tmp-dir\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:41.149662 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.149647    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-config-volume\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:41.177432 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.177403    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpx4v\" (UniqueName: \"kubernetes.io/projected/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-kube-api-access-bpx4v\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:41.178590 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.178569    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pbbk\" (UniqueName: \"kubernetes.io/projected/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-api-access-8pbbk\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.178685 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.178670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v"
Oct 30 00:04:41.185315 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.185296    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxjrt"
Oct 30 00:04:41.505684 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.505659    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v"]
Oct 30 00:04:41.506970 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:41.506945    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a3d601_1733_40ac_834e_7378e3098542.slice/crio-1df8352421062288b13431cc5b8f604af9e547c892adb48f05e6fb46a3a7234b WatchSource:0}: Error finding container 1df8352421062288b13431cc5b8f604af9e547c892adb48f05e6fb46a3a7234b: Status 404 returned error can't find the container with id 1df8352421062288b13431cc5b8f604af9e547c892adb48f05e6fb46a3a7234b
Oct 30 00:04:41.518944 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.518920    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxjrt"]
Oct 30 00:04:41.519423 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:41.519404    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb4d953_9d64_45fb_bbff_2d32e7cbdaf3.slice/crio-d58327cd41de76d79f61dc4623d0cacb78859c311a97ef27a7fba1905279caee WatchSource:0}: Error finding container d58327cd41de76d79f61dc4623d0cacb78859c311a97ef27a7fba1905279caee: Status 404 returned error can't find the container with id d58327cd41de76d79f61dc4623d0cacb78859c311a97ef27a7fba1905279caee
Oct 30 00:04:41.652309 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.652281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:41.652441 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.652336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.652441 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.652382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:41.652509 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:41.652442    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:04:41.652509 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:41.652449    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:04:41.652509 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:41.652503    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls podName:3f2407f8-44bd-4a86-af6d-66429fb7ecfd nodeName:}" failed. No retries permitted until 2026-10-30 00:04:42.652488929 +0000 UTC m=+31.676188970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls") pod "dns-default-xzfnp" (UID: "3f2407f8-44bd-4a86-af6d-66429fb7ecfd") : secret "dns-default-metrics-tls" not found
Oct 30 00:04:41.652603 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:41.652502    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:41.652603 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:41.652517    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm podName:657fcf3d-0fff-4894-94dc-2dc1b84a9faf nodeName:}" failed. No retries permitted until 2026-10-30 00:04:42.652511417 +0000 UTC m=+31.676211450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvj9n" (UID: "657fcf3d-0fff-4894-94dc-2dc1b84a9faf") : configmap "kube-rbac-proxy" not found
Oct 30 00:04:41.652603 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:41.652556    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls podName:657fcf3d-0fff-4894-94dc-2dc1b84a9faf nodeName:}" failed. No retries permitted until 2026-10-30 00:04:42.652543734 +0000 UTC m=+31.676243804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvj9n" (UID: "657fcf3d-0fff-4894-94dc-2dc1b84a9faf") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:41.760891 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.760821    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxjrt" event={"ID":"9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3","Type":"ContainerStarted","Data":"d58327cd41de76d79f61dc4623d0cacb78859c311a97ef27a7fba1905279caee"}
Oct 30 00:04:41.761947 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:41.761924    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v" event={"ID":"66a3d601-1733-40ac-834e-7378e3098542","Type":"ContainerStarted","Data":"1df8352421062288b13431cc5b8f604af9e547c892adb48f05e6fb46a3a7234b"}
Oct 30 00:04:42.435813 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:42.435777    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:42.436002 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:42.435864    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:42.436002 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:42.435964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:42.452998 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:42.452972    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:04:42.453148 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:42.453139    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thx59\""
Oct 30 00:04:42.453254 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:42.453229    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:04:42.453365 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:42.453351    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:04:42.466573 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:42.466553    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m7wf2\""
Oct 30 00:04:42.466676 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:42.466553    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-pxfs7\""
Oct 30 00:04:42.466676 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:42.466613    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:04:42.658503 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:42.658460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:42.658503 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:42.658504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:42.658975 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:42.658566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:42.658975 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:42.658659    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:42.658975 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:42.658706    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:04:42.658975 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:42.658669    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:04:42.658975 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:42.658739    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls podName:657fcf3d-0fff-4894-94dc-2dc1b84a9faf nodeName:}" failed. No retries permitted until 2026-10-30 00:04:44.658719561 +0000 UTC m=+33.682419618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvj9n" (UID: "657fcf3d-0fff-4894-94dc-2dc1b84a9faf") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:42.658975 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:42.658774    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm podName:657fcf3d-0fff-4894-94dc-2dc1b84a9faf nodeName:}" failed. No retries permitted until 2026-10-30 00:04:44.658764604 +0000 UTC m=+33.682464637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvj9n" (UID: "657fcf3d-0fff-4894-94dc-2dc1b84a9faf") : configmap "kube-rbac-proxy" not found
Oct 30 00:04:42.658975 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:42.658787    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls podName:3f2407f8-44bd-4a86-af6d-66429fb7ecfd nodeName:}" failed. No retries permitted until 2026-10-30 00:04:44.658779272 +0000 UTC m=+33.682479308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls") pod "dns-default-xzfnp" (UID: "3f2407f8-44bd-4a86-af6d-66429fb7ecfd") : secret "dns-default-metrics-tls" not found
Oct 30 00:04:43.769083 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:43.769044    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v" event={"ID":"66a3d601-1733-40ac-834e-7378e3098542","Type":"ContainerStarted","Data":"af67d5106a936b19534eb45f6e865e10fd5198eb1b2768a0812c524204d101a1"}
Oct 30 00:04:43.814719 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:43.814667    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v" podStartSLOduration=2.13632535 podStartE2EDuration="3.814647805s" totalImagesPullingTime="1.678322455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:43.814138431 +0000 UTC m=+32.837838486" watchObservedRunningTime="2026-10-30 00:04:43.814647805 +0000 UTC m=+32.838347861"
Oct 30 00:04:44.171501 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.171471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:44.171612 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:44.171588    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:04:44.171660 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.171614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:44.171660 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:44.171644    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs podName:55ce16b2-ae85-4046-83b6-0fd872bca1a8 nodeName:}" failed. No retries permitted until 2026-10-30 00:05:16.171630083 +0000 UTC m=+65.195330117 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs") pod "network-metrics-daemon-6cnnk" (UID: "55ce16b2-ae85-4046-83b6-0fd872bca1a8") : secret "metrics-daemon-secret" not found
Oct 30 00:04:44.173741 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.173715    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:44.272415 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.272393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdwl\" (UniqueName: \"kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl\") pod \"network-check-target-tr4w8\" (UID: \"a055af25-0d4a-4b69-b395-259f12a4f363\") " pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:44.274818 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.274796    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdwl\" (UniqueName: \"kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl\") pod \"network-check-target-tr4w8\" (UID: \"a055af25-0d4a-4b69-b395-259f12a4f363\") " pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:44.318590 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.318567    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:44.565475 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.565441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:44.639329 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.639306    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v5tjs"]
Oct 30 00:04:44.641364 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:44.641337    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d12dbae_7cc9_41fd_af0d_3dc152a3d6f5.slice/crio-dd029bebea728d7c75c5b56b3aeeb5be0a3cd0911e326d53486c57c9eb0068c4 WatchSource:0}: Error finding container dd029bebea728d7c75c5b56b3aeeb5be0a3cd0911e326d53486c57c9eb0068c4: Status 404 returned error can't find the container with id dd029bebea728d7c75c5b56b3aeeb5be0a3cd0911e326d53486c57c9eb0068c4
Oct 30 00:04:44.674731 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.674700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:44.674825 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.674742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:44.674825 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.674790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:44.674936 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:44.674843    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:44.674936 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:44.674878    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:04:44.674936 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:44.674902    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls podName:657fcf3d-0fff-4894-94dc-2dc1b84a9faf nodeName:}" failed. No retries permitted until 2026-10-30 00:04:48.674888676 +0000 UTC m=+37.698588757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvj9n" (UID: "657fcf3d-0fff-4894-94dc-2dc1b84a9faf") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:44.674936 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:44.674904    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:04:44.674936 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:44.674929    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm podName:657fcf3d-0fff-4894-94dc-2dc1b84a9faf nodeName:}" failed. No retries permitted until 2026-10-30 00:04:48.674916675 +0000 UTC m=+37.698616709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvj9n" (UID: "657fcf3d-0fff-4894-94dc-2dc1b84a9faf") : configmap "kube-rbac-proxy" not found
Oct 30 00:04:44.675124 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:44.674945    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls podName:3f2407f8-44bd-4a86-af6d-66429fb7ecfd nodeName:}" failed. No retries permitted until 2026-10-30 00:04:48.674939136 +0000 UTC m=+37.698639168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls") pod "dns-default-xzfnp" (UID: "3f2407f8-44bd-4a86-af6d-66429fb7ecfd") : secret "dns-default-metrics-tls" not found
Oct 30 00:04:44.675914 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.675901    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jvrpq_41e90ea1-6a7c-4007-89f4-bba585722b8a/dns-node-resolver/0.log"
Oct 30 00:04:44.772047 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.772006    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxjrt" event={"ID":"9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3","Type":"ContainerStarted","Data":"927b73c31d5a4c1f52c3337ac3130c870751a03aa856034e2b4bdd7ab039ba5c"}
Oct 30 00:04:44.773229 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.773209    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v5tjs" event={"ID":"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5","Type":"ContainerStarted","Data":"dd029bebea728d7c75c5b56b3aeeb5be0a3cd0911e326d53486c57c9eb0068c4"}
Oct 30 00:04:44.821241 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.821200    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qxjrt" podStartSLOduration=2.047305427 podStartE2EDuration="4.821187329s" totalImagesPullingTime="2.773881902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:44.820632124 +0000 UTC m=+33.844332179" watchObservedRunningTime="2026-10-30 00:04:44.821187329 +0000 UTC m=+33.844887382"
Oct 30 00:04:44.874266 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:44.874248    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tr4w8"]
Oct 30 00:04:44.874795 ip-10-0-137-69 kubenswrapper[2581]: W1030 00:04:44.874777    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda055af25_0d4a_4b69_b395_259f12a4f363.slice/crio-cbc65ccf155499c12572b2e5e29ebe24480a0390210a4982ec9239b1373edfa1 WatchSource:0}: Error finding container cbc65ccf155499c12572b2e5e29ebe24480a0390210a4982ec9239b1373edfa1: Status 404 returned error can't find the container with id cbc65ccf155499c12572b2e5e29ebe24480a0390210a4982ec9239b1373edfa1
Oct 30 00:04:45.280593 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:45.280566    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6hq6g_40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f/node-ca/0.log"
Oct 30 00:04:45.778017 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:45.777973    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tr4w8" event={"ID":"a055af25-0d4a-4b69-b395-259f12a4f363","Type":"ContainerStarted","Data":"cbc65ccf155499c12572b2e5e29ebe24480a0390210a4982ec9239b1373edfa1"}
Oct 30 00:04:46.275897 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:46.275839    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qxjrt_9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3/serve-healthcheck-canary/0.log"
Oct 30 00:04:46.683732 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:46.683661    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-4jj6v_66a3d601-1733-40ac-834e-7378e3098542/migrator/0.log"
Oct 30 00:04:48.704366 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:48.704327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:48.704812 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:48.704395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:48.704812 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:48.704457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:48.704812 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:48.704569    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:48.704812 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:48.704579    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:04:48.704812 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:48.704642    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls podName:657fcf3d-0fff-4894-94dc-2dc1b84a9faf nodeName:}" failed. No retries permitted until 2026-10-30 00:04:56.704623501 +0000 UTC m=+45.728323548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvj9n" (UID: "657fcf3d-0fff-4894-94dc-2dc1b84a9faf") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:48.704812 ip-10-0-137-69 kubenswrapper[2581]: E1030 00:04:48.704662    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls podName:3f2407f8-44bd-4a86-af6d-66429fb7ecfd nodeName:}" failed. No retries permitted until 2026-10-30 00:04:56.704651543 +0000 UTC m=+45.728351578 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls") pod "dns-default-xzfnp" (UID: "3f2407f8-44bd-4a86-af6d-66429fb7ecfd") : secret "dns-default-metrics-tls" not found
Oct 30 00:04:48.705117 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:48.704932    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:48.787705 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:48.787680    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tr4w8" event={"ID":"a055af25-0d4a-4b69-b395-259f12a4f363","Type":"ContainerStarted","Data":"71391d632fcfb5a907c1dc61ecd03bc0fc32ee87a6a167f9204e5c450410ef55"}
Oct 30 00:04:48.787847 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:48.787821    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:48.831259 ip-10-0-137-69 kubenswrapper[2581]: I1030 00:04:48.831212    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tr4w8" podStartSLOduration=34.508089755 podStartE2EDuration="37.831197034s" totalImagesPullingTime="3.323107279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:48.830269107 +0000 UTC m=+37.853969161" watchObservedRunningTime="2026-10-30 00:04:48.831197034 +0000 UTC m=+37.854897090"
Oct 30 00:04:50.416631 ip-10-0-137-69 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 00:04:50.432390 ip-10-0-137-69 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 00:04:50.432553 ip-10-0-137-69 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 00:04:50.432599 ip-10-0-137-69 systemd[1]: kubelet.service: Consumed 2.147s CPU time.
Oct 30 00:04:50.446701 ip-10-0-137-69 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:04:50.526845 ip-10-0-137-69 kubenswrapper[4441]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:04:50.526845 ip-10-0-137-69 kubenswrapper[4441]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:04:50.526845 ip-10-0-137-69 kubenswrapper[4441]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:04:50.526845 ip-10-0-137-69 kubenswrapper[4441]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:04:50.528974 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.528902    4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:50.528974 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.528974    4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.528978    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.528981    4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.528984    4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.528988    4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.528990    4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.528993    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.528996    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.528998    4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529001    4441 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529004    4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529007    4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529011    4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529014    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529018    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529034    4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529038    4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529041    4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529043    4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:50.529057 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529046    4441 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529049    4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529051    4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529054    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529056    4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529059    4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529062    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529065    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529068    4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529071    4441 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529073    4441 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529076    4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529080    4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529082    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529085    4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529087    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529090    4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529092    4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529095    4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529097    4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:50.529529 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529100    4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529102    4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529105    4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529107    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529109    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529112    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529114    4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529117    4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529119    4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529122    4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529124    4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529127    4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529130    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529132    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529135    4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529138    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529140    4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529143    4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529145    4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529148    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:50.530019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529150    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529153    4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529155    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529158    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529160    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529163    4441 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529166    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529168    4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529172    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529174    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529177    4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529180    4441 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529182    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529185    4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529187    4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529189    4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529192    4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529195    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529198    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:50.530528 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529200    4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529203    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529205    4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529208    4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529210    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529213    4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529215    4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529218    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529221    4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529224    4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529226    4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529229    4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529232    4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529234    4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529237    4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529241    4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529246    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529250    4441 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529252    4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:50.530998 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529256    4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529259    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529261    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529264    4441 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529268    4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529270    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529273    4441 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.529275    4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530078    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530087    4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530090    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530093    4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530095    4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530098    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530101    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530103    4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530106    4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530109    4441 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530112    4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:50.531490 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530115    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530118    4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530121    4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530124    4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530126    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530129    4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530132    4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530134    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530137    4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530140    4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530143    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530145    4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530148    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530151    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530155    4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530158    4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530160    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530163    4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530166    4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:50.531951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530168    4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530171    4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530173    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530176    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530179    4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530181    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530184    4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530187    4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530191    4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530194    4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530197    4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530200    4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530202    4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530205    4441 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530208    4441 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530211    4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530213    4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530216    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530219    4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:50.532459 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530222    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530224    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530227    4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530229    4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530232    4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530235    4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530238    4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530240    4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530243    4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530246    4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530248    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530251    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530254    4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530256    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530258    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530261    4441 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530264    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530267    4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530269    4441 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530272    4441 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:50.532924 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530275    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530277    4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530280    4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530283    4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530285    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530288    4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530290    4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530294    4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530296    4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530299    4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530301    4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530304    4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530308    4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530311    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530314    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530316    4441 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530319    4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530322    4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530325    4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:50.533496 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530327    4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530330    4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530332    4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530335    4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530338    4441 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530340    4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530342    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530345    4441 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530352    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530355    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530357    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530360    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530362    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530365    4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530367    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530370    4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530372    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530375    4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530847    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530852    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:50.533966 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530855    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530859    4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530862    4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530864    4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530867    4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530869    4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530873    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530875    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530878    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530881    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530884    4441 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530886    4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530889    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530891    4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530894    4441 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530897    4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530900    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530902    4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530905    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:50.534501 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530907    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530910    4441 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530912    4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530915    4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530918    4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530920    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530923    4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530925    4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530928    4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530930    4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530933    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530935    4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530938    4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530941    4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530944    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530946    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530948    4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530951    4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530954    4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530957    4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:50.535199 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530959    4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530962    4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530964    4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530968    4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530971    4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530974    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530977    4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530980    4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530983    4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530985    4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530988    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530992    4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.530996    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531000    4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531005    4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531009    4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531013    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531016    4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531019    4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:50.535832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531034    4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531038    4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531041    4441 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531043    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531046    4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531049    4441 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531052    4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531057    4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531061    4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531064    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531066    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531069    4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531071    4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531074    4441 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531076    4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531079    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531081    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531084    4441 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531086    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:50.536327 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531089    4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531092    4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531094    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531097    4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531101    4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531103    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531106    4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531109    4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531112    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531114    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531117    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531121    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531124    4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531127    4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531129    4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531132    4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531136    4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531138    4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531141    4441 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:50.536795 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531143    4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531146    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531148    4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531151    4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531154    4441 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531156    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531158    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531161    4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531249    4441 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531257    4441 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531263    4441 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531267    4441 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531272    4441 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531275    4441 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531280    4441 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531284    4441 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531287    4441 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531290    4441 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531294    4441 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531298    4441 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531301    4441 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531304    4441 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531307    4441 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:04:50.537273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531310    4441 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531314    4441 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531317    4441 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531322    4441 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531325    4441 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531328    4441 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531331    4441 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531335    4441 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531338    4441 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531341    4441 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531345    4441 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531348    4441 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531351    4441 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531354    4441 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531357    4441 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531360    4441 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531363    4441 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531367    4441 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531371    4441 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531374    4441 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531377    4441 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531380    4441 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531382    4441 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531386    4441 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531390    4441 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:04:50.537834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531393    4441 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531395    4441 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531398    4441 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531403    4441 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531406    4441 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531409    4441 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531412    4441 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531415    4441 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531419    4441 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531422    4441 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531425    4441 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531428    4441 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531431    4441 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531434    4441 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531438    4441 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531441    4441 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531444    4441 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531447    4441 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531450    4441 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531453    4441 flags.go:64] FLAG: --help="false"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531456    4441 flags.go:64] FLAG: --hostname-override="ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531459    4441 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531462    4441 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:04:50.538458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531465    4441 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531468    4441 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531471    4441 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531474    4441 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531477    4441 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531480    4441 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531485    4441 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531488    4441 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531491    4441 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531494    4441 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531497    4441 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531501    4441 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531504    4441 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531507    4441 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531510    4441 flags.go:64] FLAG: --lock-file=""
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531513    4441 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531516    4441 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531520    4441 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531526    4441 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531528    4441 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531532    4441 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531535    4441 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531537    4441 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531541    4441 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:04:50.539063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531544    4441 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531546    4441 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531551    4441 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531554    4441 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531557    4441 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531560    4441 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531563    4441 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531566    4441 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531569    4441 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531572    4441 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531575    4441 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531578    4441 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531584    4441 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531588    4441 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531592    4441 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531595    4441 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531598    4441 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531601    4441 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531604    4441 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531607    4441 flags.go:64] FLAG: --port="10250"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531610    4441 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531613    4441 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02d2468df5f0482ca"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531616    4441 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531619    4441 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531622    4441 flags.go:64] FLAG: --register-node="true"
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531627    4441 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:04:50.539746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531632    4441 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531636    4441 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531639    4441 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531641    4441 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531645    4441 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531648    4441 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531651    4441 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531654    4441 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531657    4441 flags.go:64] FLAG: --runonce="false"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531659    4441 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531662    4441 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531666    4441 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531668    4441 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531671    4441 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531675    4441 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531678    4441 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531681    4441 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531683    4441 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531686    4441 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531689    4441 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531693    4441 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531696    4441 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531699    4441 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531702    4441 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531708    4441 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:04:50.540581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531711    4441 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531714    4441 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531718    4441 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531721    4441 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531724    4441 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531727    4441 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531731    4441 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531734    4441 flags.go:64] FLAG: --v="2"
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531739    4441 flags.go:64] FLAG: --version="false"
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531743    4441 flags.go:64] FLAG: --vmodule=""
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531747    4441 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.531750    4441 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531879    4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531886    4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531889    4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531892    4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531894    4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531897    4441 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531900    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531902    4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531905    4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531907    4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531910    4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:50.541372 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531913    4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531915    4441 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531918    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531920    4441 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531924    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531927    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531929    4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531932    4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531934    4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531937    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531940    4441 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531942    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531945    4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531947    4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531951    4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531955    4441 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531958    4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531961    4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531964    4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531966    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:50.542319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531969    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531971    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531974    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531977    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531979    4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531982    4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531984    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531987    4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531990    4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531992    4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531995    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.531998    4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532000    4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532003    4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532005    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532008    4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532012    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532015    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532017    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:50.543019 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532020    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532036    4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532041    4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532044    4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532047    4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532049    4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532052    4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532054    4441 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532059    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532062    4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532065    4441 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532067    4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532070    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532072    4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532075    4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532077    4441 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532080    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532082    4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532085    4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532087    4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:50.543525 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532090    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532093    4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532096    4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532098    4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532100    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532103    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532106    4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532108    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532112    4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532117    4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532121    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532123    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532126    4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532129    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532132    4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532134    4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532137    4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532139    4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532142    4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:50.544155 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532144    4441 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532149    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532152    4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532154    4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532157    4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532160    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532162    4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532166    4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532170    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532173    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532175    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532178    4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532181    4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532184    4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532186    4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532189    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:50.544701 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532191    4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.532218    4441 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532255    4441 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532260    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532263    4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532266    4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532269    4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532273    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532276    4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532278    4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532281    4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532284    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532286    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532289    4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532292    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532296    4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:50.545265 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532299    4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532303    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532306    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532309    4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532312    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532315    4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532317    4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532320    4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532323    4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532325    4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532328    4441 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532330    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532333    4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532335    4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532338    4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532340    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532343    4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532358    4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532362    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:50.545657 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532365    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532368    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532371    4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532374    4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532378    4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532381    4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532383    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532386    4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532390    4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532393    4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532395    4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532398    4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532401    4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532403    4441 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532407    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532410    4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532412    4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532415    4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532418    4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532421    4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:50.546127 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532423    4441 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532426    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532428    4441 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532431    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532434    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532436    4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532439    4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532441    4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532444    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532447    4441 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532449    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532452    4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532454    4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532457    4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532459    4441 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532462    4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532466    4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532469    4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532472    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532474    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:50.546615 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532477    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532479    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532482    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532484    4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532487    4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532489    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532493    4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532495    4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532498    4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532501    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532504    4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532507    4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532509    4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532512    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532514    4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532516    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532519    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532521    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532525    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:50.547157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532528    4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532530    4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532533    4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532535    4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532538    4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532540    4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532543    4441 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532545    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532548    4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532551    4441 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532554    4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532557    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532559    4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.532562    4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.536972    4441 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.536988    4441 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.537006    4441 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.537011    4441 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537087    4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:50.547624 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537093    4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537096    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537100    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537103    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537105    4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537108    4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537111    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537114    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537117    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537119    4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537122    4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537125    4441 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537133    4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537138    4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537142    4441 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537146    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537149    4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537152    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537155    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:50.548108 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537157    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537160    4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537163    4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537166    4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537169    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537171    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537174    4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537177    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537179    4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537182    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537184    4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537187    4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537190    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537193    4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537195    4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537198    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537200    4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537203    4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537205    4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537208    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:50.548585 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537211    4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537213    4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537216    4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537218    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537221    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537224    4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537227    4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537230    4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537233    4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537236    4441 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537238    4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537241    4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537244    4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537247    4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537249    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537252    4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537255    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537257    4441 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537260    4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537263    4441 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:50.549253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537266    4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537268    4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537271    4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537274    4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537277    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537280    4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537282    4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537285    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537288    4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537290    4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537293    4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537295    4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537298    4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537301    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537303    4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537306    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537308    4441 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537311    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537314    4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:50.549897 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537317    4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537320    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537323    4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537326    4441 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537329    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537331    4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537334    4441 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537337    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537340    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537342    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537345    4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537348    4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537350    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537353    4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537355    4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537358    4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537361    4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537363    4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537366    4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537368    4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:50.550607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537371    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537374    4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537376    4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537379    4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537381    4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537384    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537386    4441 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.537414    4441 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537544    4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537549    4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537552    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537555    4441 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537557    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537560    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537563    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537566    4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:50.551443 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537569    4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537572    4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537575    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537578    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537581    4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537583    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537586    4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537589    4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537591    4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537594    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537598    4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537602    4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537605    4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537608    4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537611    4441 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537614    4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537617    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537620    4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537622    4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537625    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:50.551985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537627    4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537630    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537632    4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537635    4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537638    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537641    4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537645    4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537648    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537651    4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537654    4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537657    4441 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537659    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537662    4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537665    4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537667    4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537670    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537673    4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537675    4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537678    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:50.552524 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537680    4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537683    4441 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537686    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537688    4441 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537691    4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537693    4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537696    4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537698    4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537701    4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537703    4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537706    4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537708    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537711    4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537714    4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537716    4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537719    4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537721    4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537724    4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537726    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537729    4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:50.553010 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537731    4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537734    4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537736    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537739    4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537742    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537745    4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537748    4441 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537750    4441 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537753    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537756    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537758    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537761    4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537763    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537766    4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537768    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537771    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537773    4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537776    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537778    4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:50.553538 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537781    4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537783    4441 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537786    4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537788    4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537791    4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537793    4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537796    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537798    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537801    4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537803    4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537806    4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537808    4441 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537811    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537813    4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537816    4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537818    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537821    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537824    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537829    4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:50.554055 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537832    4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.537855    4441 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537885    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537889    4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537892    4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537895    4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537898    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537901    4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537903    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537906    4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537908    4441 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537911    4441 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537914    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537916    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537919    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537922    4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:04:50.554546 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537924    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537927    4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537929    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537932    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537934    4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537937    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537940    4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537942    4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537945    4441 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537947    4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537950    4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537952    4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537955    4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537958    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537961    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537965    4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537968    4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537971    4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537974    4441 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:04:50.554953 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537976    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537979    4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537981    4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537984    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537987    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537989    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537992    4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537994    4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537997    4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.537999    4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538002    4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538005    4441 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538007    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538010    4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538012    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538015    4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538017    4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538020    4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538036    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538039    4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:04:50.555483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538042    4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538044    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538047    4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538050    4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538052    4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538055    4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538059    4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538062    4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538067    4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538070    4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538074    4441 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538077    4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538080    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538083    4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538086    4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538088    4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538091    4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538093    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538096    4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:04:50.555965 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538098    4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538101    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538105    4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538109    4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538112    4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538115    4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538118    4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538120    4441 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538123    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538125    4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538128    4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538130    4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538132    4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538135    4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538137    4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538140    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538142    4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538145    4441 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538148    4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:04:50.556454 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538150    4441 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538154    4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538158    4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538160    4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538163    4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538166    4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538168    4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538171    4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538173    4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538176    4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538178    4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538181    4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538183    4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538186    4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:50.538188    4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.538241    4441 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.539169    4441 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.539224    4441 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.539409    4441 server.go:1014] "Starting client certificate rotation"
Oct 30 00:04:50.556900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.539564    4441 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:04:50.557498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.539628    4441 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 23:59:11 +0000 UTC" deadline="2028-07-30 21:36:09.331460998 +0000 UTC"
Oct 30 00:04:50.557498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.539667    4441 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15357h31m18.791797136s"
Oct 30 00:04:50.557498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.539874    4441 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:04:50.557498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.540899    4441 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:04:50.557498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.542489    4441 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:04:50.557498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.543948    4441 log.go:25] "Validated CRI v1 image API"
Oct 30 00:04:50.557498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.544713    4441 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:04:50.557498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.546427    4441 fs.go:135] Filesystem UUIDs: map[12b6fd48-b32b-4065-a06c-c11ead05fb55:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 eee7c887-bd34-4bff-a0e3-52bb7a2cd80a:/dev/nvme0n1p3]
Oct 30 00:04:50.557770 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.546446    4441 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0da529af890d1e92daecf5aa5bf2521476c5588c65bd93615e3a4ed7bbea87b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0da529af890d1e92daecf5aa5bf2521476c5588c65bd93615e3a4ed7bbea87b7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f3dc17f2583b259fa05ecb1692f18956c129b0b91ff07ed5bdabe60dff5c0c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f3dc17f2583b259fa05ecb1692f18956c129b0b91ff07ed5bdabe60dff5c0c0/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1df8352421062288b13431cc5b8f604af9e547c892adb48f05e6fb46a3a7234b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1df8352421062288b13431cc5b8f604af9e547c892adb48f05e6fb46a3a7234b/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/241ece0fb72045fadb95614504bdf396f0149afbb4f0e9b3b8913f3bc43c83f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/241ece0fb72045fadb95614504bdf396f0149afbb4f0e9b3b8913f3bc43c83f7/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3035c3dee7124a74822917e8cb6ae4859c1b34e56a6e7d1ac72eed640c4881ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3035c3dee7124a74822917e8cb6ae4859c1b34e56a6e7d1ac72eed640c4881ad/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59edc93c744b47b3b583bf3add9433bad217423f019c2e57e96c6a8c467214af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59edc93c744b47b3b583bf3add9433bad217423f019c2e57e96c6a8c467214af/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73d77211ae7474d78bf3f1de0e8300480f343b294d4b0c6e684126fb3dc0ebf6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73d77211ae7474d78bf3f1de0e8300480f343b294d4b0c6e684126fb3dc0ebf6/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ab4c5cff26f0ff20dffb1a927899c05e18dc4adbe4c66ea359d9010845a0e7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ab4c5cff26f0ff20dffb1a927899c05e18dc4adbe4c66ea359d9010845a0e7f/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b78005a723fff44b123d2bb9e6344b7175cd654ce789fcced15b22370bfbb7c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b78005a723fff44b123d2bb9e6344b7175cd654ce789fcced15b22370bfbb7c6/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbc65ccf155499c12572b2e5e29ebe24480a0390210a4982ec9239b1373edfa1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbc65ccf155499c12572b2e5e29ebe24480a0390210a4982ec9239b1373edfa1/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d417a4f131c626416459d24c725bf4c0cc4c42aa1bfa316cd4e02c6c6f2ecddb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d417a4f131c626416459d24c725bf4c0cc4c42aa1bfa316cd4e02c6c6f2ecddb/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d58327cd41de76d79f61dc4623d0cacb78859c311a97ef27a7fba1905279caee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d58327cd41de76d79f61dc4623d0cacb78859c311a97ef27a7fba1905279caee/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd029bebea728d7c75c5b56b3aeeb5be0a3cd0911e326d53486c57c9eb0068c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd029bebea728d7c75c5b56b3aeeb5be0a3cd0911e326d53486c57c9eb0068c4/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f72d6d823f592be1cab0e2a8af30b7bf158bc2a39b11875d74a6a44a0dbb1ae6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f72d6d823f592be1cab0e2a8af30b7bf158bc2a39b11875d74a6a44a0dbb1ae6/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe/volumes/kubernetes.io~projected/kube-api-access-rffhf:{mountpoint:/var/lib/kubelet/pods/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe/volumes/kubernetes.io~projected/kube-api-access-rffhf major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23960d87-bf90-43c3-9ee6-464afa4c42ea/volumes/kubernetes.io~projected/kube-api-access-52b7g:{mountpoint:/var/lib/kubelet/pods/23960d87-bf90-43c3-9ee6-464afa4c42ea/volumes/kubernetes.io~projected/kube-api-access-52b7g major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f2407f8-44bd-4a86-af6d-66429fb7ecfd/volumes/kubernetes.io~projected/kube-api-access-bpx4v:{mountpoint:/var/lib/kubelet/pods/3f2407f8-44bd-4a86-af6d-66429fb7ecfd/volumes/kubernetes.io~projected/kube-api-access-bpx4v major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f/volumes/kubernetes.io~projected/kube-api-access-hw6bf:{mountpoint:/var/lib/kubelet/pods/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f/volumes/kubernetes.io~projected/kube-api-access-hw6bf major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41e90ea1-6a7c-4007-89f4-bba585722b8a/volumes/kubernetes.io~projected/kube-api-access-8mtkj:{mountpoint:/var/lib/kubelet/pods/41e90ea1-6a7c-4007-89f4-bba585722b8a/volumes/kubernetes.io~projected/kube-api-access-8mtkj major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55ce16b2-ae85-4046-83b6-0fd872bca1a8/volumes/kubernetes.io~projected/kube-api-access-4f9bp:{mountpoint:/var/lib/kubelet/pods/55ce16b2-ae85-4046-83b6-0fd872bca1a8/volumes/kubernetes.io~projected/kube-api-access-4f9bp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/657fcf3d-0fff-4894-94dc-2dc1b84a9faf/volumes/kubernetes.io~projected/kube-api-access-8pbbk:{mountpoint:/var/lib/kubelet/pods/657fcf3d-0fff-4894-94dc-2dc1b84a9faf/volumes/kubernetes.io~projected/kube-api-access-8pbbk major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66a3d601-1733-40ac-834e-7378e3098542/volumes/kubernetes.io~projected/kube-api-access-nwjfl:{mountpoint:/var/lib/kubelet/pods/66a3d601-1733-40ac-834e-7378e3098542/volumes/kubernetes.io~projected/kube-api-access-nwjfl major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/854339cf-9de2-4d3f-9275-20cb99cb7c89/volumes/kubernetes.io~projected/kube-api-access-jv7zp:{mountpoint:/var/lib/kubelet/pods/854339cf-9de2-4d3f-9275-20cb99cb7c89/volumes/kubernetes.io~projected/kube-api-access-jv7zp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d2ad09d-f117-46ba-bf91-9dbcccca5748/volumes/kubernetes.io~projected/kube-api-access-2wtc6:{mountpoint:/var/lib/kubelet/pods/8d2ad09d-f117-46ba-bf91-9dbcccca5748/volumes/kubernetes.io~projected/kube-api-access-2wtc6 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3/volumes/kubernetes.io~projected/kube-api-access-xxq4w:{mountpoint:/var/lib/kubelet/pods/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3/volumes/kubernetes.io~projected/kube-api-access-xxq4w major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a055af25-0d4a-4b69-b395-259f12a4f363/volumes/kubernetes.io~projected/kube-api-access-ltdwl:{mountpoint:/var/lib/kubelet/pods/a055af25-0d4a-4b69-b395-259f12a4f363/volumes/kubernetes.io~projected/kube-api-access-ltdwl major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf392012-ba7c-4999-b0fe-f0ae7cee60f7/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/bf392012-ba7c-4999-b0fe-f0ae7cee60f7/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccef4fa8-30d6-4e12-b2f1-470b13955cc5/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ccef4fa8-30d6-4e12-b2f1-470b13955cc5/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccef4fa8-30d6-4e12-b2f1-470b13955cc5/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ccef4fa8-30d6-4e12-b2f1-470b13955cc5/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccef4fa8-30d6-4e12-b2f1-470b13955cc5/volumes/kubernetes.io~projected/kube-api-access-gpklh:{mountpoint:/var/lib/kubelet/pods/ccef4fa8-30d6-4e12-b2f1-470b13955cc5/volumes/kubernetes.io~projected/kube-api-access-gpklh major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d74677bd-f7f3-4a55-b5e7-0216819f5400/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d74677bd-f7f3-4a55-b5e7-0216819f5400/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d74677bd-f7f3-4a55-b5e7-0216819f5400/volumes/kubernetes.io~projected/kube-api-access-7xwzm:{mountpoint:/var/lib/kubelet/pods/d74677bd-f7f3-4a55-b5e7-0216819f5400/volumes/kubernetes.io~projected/kube-api-access-7xwzm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d74677bd-f7f3-4a55-b5e7-0216819f5400/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d74677bd-f7f3-4a55-b5e7-0216819f5400/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4c9ebd672ffd7f950033a3fa480422c33d3e4bcfd5d4252d794536c12667bdbf/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a89794d2ffa144b367479f714fb7cddc2ac207375e63fb953a3a67969ffabfed/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5cfb5ac49bc5e5fb46d66c7ed474576cdf0908f153f5af9883181f6c6aeaf9ef/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/a6eed79be5f92c09a8038a6e060b92ed802b9ef61e78adac9d0b6553d27cd10c/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6844dbfd1f8297f755d2e1780b6c092b20f852f022026dce2db2f9c846fb6db7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a1fb08b76fd0ab2c7c42bf1d250efa1eb4f38f43d3d89d8dc488755f656da317/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e62e5459cb8c3f369152a71cbf569e466c0989621f2b401da5a0867e8343b4c4/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c93ca3436999122ea96c17bf94b820162dcc11fd59334e92137dcdf668594d13/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ef07d43c03ef9c83cec69cf163a92e395d53801e757c0cff6d7a9450651719ee/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e0306a1e2d08163cc9e17b68aa932a7aecb3c9063f1da19dc239a7f175ec0814/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/74656030d9ee310f84ee521a839dda898190ec97b9f8a85c8d3df84f856a2269/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/b8bd63b7d5c52124298db2137e4d08898eb351ceb6c5f7069fe9ffcebc8d3e1f/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8264ae70da94423e1b671c56e594a528ad87ae97efb1e3c1d999b3f1a7779ffa/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d3f7aa6afda6757ace99ac8fee9b42a7c4f1c0c8b0a55818514be9e4ad305f88/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/8014874495de361daf87f6f2859d3b1c4e8afe57c791ce79282cbea540a7f452/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/74b7b12359a9985b2cb94a19806f42edad512707f6506d9676a7cee9d5db61a0/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/bc78ca6d9517a5f12fc87dc283e3221f9976006600e5be59d02592efd3e2b4b1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4a87ea469ba03accfe7b0020d0845f4f7f7ec711dfa59ff18782f0d31f25bf1f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/9667945669e6caf03f1f1c442d0cd61e4203a9ea69288f4d212744d286e7bcfe/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e5a4d68b7cc011342c799f1763546ffb637387cb8557b3d6ec17a69b1e41a399/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2b7c47a22c19d439d6a96993e159b305687b7d4e5cbf2467b7180615e04972d9/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/b69e3004597a90468235c6e46eb27047979618cb4b858fe37799798219336802/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/09ccec9be650672adf82263183bcd9c33da71a262bf47459175979da14b7b10c/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/893ea96281b926b8c6ba4b6b1576d7cd113c0b914a0856d94636e455b64653db/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/10d0a0e3db27104de85770c8d0198b9776c15533a3f7b192dd02ca065c0e143d/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/7cbf4594737e34bfa341435c367f5fe5850b2084a4b7f3e3683a5e25e17e0d46/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/6586b15e3c3f6f52c3600dabfdd7b3be1381f8d613a10c9c1732e74579c0477e/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/aab73c1c3956be062be51490d0340f673c023b63894445d85ef2d1117c74f8e4/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/7dc088ca90dcc62f67d4fa98a5aba6e8e44a9126824b20641c5892530fda0752/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/e6ce7c4f9b9705668a74988abfd2fafc9c7e4b54aada2623351e51a914170ecc/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0d71cd7cf1bfbe447c05d85560b201dc8b09090184d32774feb881b2d3bacdb7/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2c21d0937b4607ec6d8088fd5f4088c57b3fb72366ced1e5288c65f587117b04/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/897857b928df5445cbbfa8ff6bde60e5df27847448c69c3b889e7ed473ef06e6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/31203ced267d8d0ca0986582425302c7b4342287b1596e3bc4d1024136f9fc85/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/83c9aad8a68aa80f143916731a8572a86146f140948abbe8dddd0e95a44b5a1a/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d7a9bc7d4ed5ee15c43e34c15a487e86720467e720575f398625aa27aed4113d/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ed0d61198272febe91ac7d454d0453a0edec6a654de95269fe79e7aeb34df3e4/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5f386292163618e15bc37e8286363dd9d2f041e847d6f78665502e7d5174d48d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/31129a2836bb701f2a837eeb0340838e71765a45300a944cabca36bf62819c17/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.552543    4441 manager.go:217] Machine: {Timestamp:2026-10-30 00:04:50.551760799 +0000 UTC m=+0.077054669 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101236 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20d383c00d4a614a6b4b705de3ef75 SystemUUID:ec20d383-c00d-4a61-4a6b-4b705de3ef75 BootID:bcda9035-2774-4980-bbc4-123d4db62cd1 Filesystems:[{Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ab4c5cff26f0ff20dffb1a927899c05e18dc4adbe4c66ea359d9010845a0e7f/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/55ce16b2-ae85-4046-83b6-0fd872bca1a8/volumes/kubernetes.io~projected/kube-api-access-4f9bp DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/657fcf3d-0fff-4894-94dc-2dc1b84a9faf/volumes/kubernetes.io~projected/kube-api-access-8pbbk DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59edc93c744b47b3b583bf3add9433bad217423f019c2e57e96c6a8c467214af/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/241ece0fb72045fadb95614504bdf396f0149afbb4f0e9b3b8913f3bc43c83f7/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccef4fa8-30d6-4e12-b2f1-470b13955cc5/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf392012-ba7c-4999-b0fe-f0ae7cee60f7/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f2407f8-44bd-4a86-af6d-66429fb7ecfd/volumes/kubernetes.io~projected/kube-api-access-bpx4v DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe/volumes/kubernetes.io~projected/kube-api-access-rffhf DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f/volumes/kubernetes.io~projected/kube-api-access-hw6bf DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/23960d87-bf90-43c3-9ee6-464afa4c42ea/volumes/kubernetes.io~projected/kube-api-access-52b7g DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3035c3dee7124a74822917e8cb6ae4859c1b34e56a6e7d1ac72eed640c4881ad/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73d77211ae7474d78bf3f1de0e8300480f343b294d4b0c6e684126fb3dc0ebf6/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a055af25-0d4a-4b69-b395-259f12a4f363/volumes/kubernetes.io~projected/kube-api-access-ltdwl DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d2ad09d-f117-46ba-bf91-9dbcccca5748/volumes/kubernetes.io~projected/kube-api-access-2wtc6 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0da529af890d1e92daecf5aa5bf2521476c5588c65bd93615e3a4ed7bbea87b7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d74677bd-f7f3-4a55-b5e7-0216819f5400/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d74677bd-f7f3-4a55-b5e7-0216819f5400/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/854339cf-9de2-4d3f-9275-20cb99cb7c89/volumes/kubernetes.io~projected/kube-api-access-jv7zp DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f3dc17f2583b259fa05ecb1692f18956c129b0b91ff07ed5bdabe60dff5c0c0/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbc65ccf155499c12572b2e5e29ebe24480a0390210a4982ec9239b1373edfa1/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d74677bd-f7f3-4a55-b5e7-0216819f5400/volumes/kubernetes.io~projected/kube-api-access-7xwzm DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1df8352421062288b13431cc5b8f604af9e547c892adb48f05e6fb46a3a7234b/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b78005a723fff44b123d2bb9e6344b7175cd654ce789fcced15b22370bfbb7c6/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f72d6d823f592be1cab0e2a8af30b7bf158bc2a39b11875d74a6a44a0dbb1ae6/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d417a4f131c626416459d24c725bf4c0cc4c42aa1bfa316cd4e02c6c6f2ecddb/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd029bebea728d7c75c5b56b3aeeb5be0a3cd0911e326d53486c57c9eb0068c4/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ccef4fa8-30d6-4e12-b2f1-470b13955cc5/volumes/kubernetes.io~projected/kube-api-access-gpklh DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3/volumes/kubernetes.io~projected/kube-api-access-xxq4w DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/66a3d601-1733-40ac-834e-7378e3098542/volumes/kubernetes.io~projected/kube-api-access-nwjfl DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d58327cd41de76d79f61dc4623d0cacb78859c311a97ef27a7fba1905279caee/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccef4fa8-30d6-4e12-b2f1-470b13955cc5/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/41e90ea1-6a7c-4007-89f4-bba585722b8a/volumes/kubernetes.io~projected/kube-api-access-8mtkj DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1df835242106228 MacAddress:72:bc:2e:79:fb:e2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c5:cc:2a:c1 Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:57:c5:3c:ab:05 Speed:0 Mtu:8901} {Name:cbc65ccf155499c MacAddress:e2:f1:93:95:7f:b3 Speed:10000 Mtu:8901} {Name:d58327cd41de76d MacAddress:6e:5d:ec:d9:98:eb Speed:10000 Mtu:8901} {Name:dd029bebea728d7 MacAddress:6a:00:49:f9:fa:96 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c5:cc:2a:c1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:a6:e9:80:c0:1a:85 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f6:f2:6e:4d:97:b6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.552766    4441 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.552803    4441 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553076    4441 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553092    4441 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-69.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553211    4441 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553219    4441 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553229    4441 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553247    4441 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553323    4441 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553458    4441 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553560    4441 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553591    4441 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553600    4441 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553609    4441 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553617    4441 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.553625    4441 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.554169    4441 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.554179    4441 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.554675    4441 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556465    4441 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556475    4441 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556564    4441 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556728    4441 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556754    4441 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556833    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556844    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556851    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556857    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556863    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556868    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556874    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556879    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556885    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556891    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556905    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556913    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556939    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:04:50.557911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.556944    4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:04:50.559982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.559967    4441 server.go:1263] "Started kubelet"
Oct 30 00:04:50.560087 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.560053    4441 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:04:50.560487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.560120    4441 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:04:50.560600 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.560516    4441 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:04:50.560523 ip-10-0-137-69 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:04:50.560808 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.560788    4441 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:04:50.561306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.561274    4441 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:04:50.564035 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.562606    4441 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:04:50.564035 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.562628    4441 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:04:50.564035 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.562693    4441 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:59:12 +0000 UTC" deadline="2028-08-04 22:39:18.629846264 +0000 UTC"
Oct 30 00:04:50.564035 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.562721    4441 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15478h34m28.067128034s"
Oct 30 00:04:50.564035 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.562731    4441 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:04:50.564035 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.562749    4441 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:04:50.564035 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.562861    4441 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:04:50.564035 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:50.562734    4441 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-69.ec2.internal\" not found"
Oct 30 00:04:50.564035 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.563850    4441 factory.go:55] Registering systemd factory
Oct 30 00:04:50.564035 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.563864    4441 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:04:50.568949 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.565050    4441 factory.go:153] Registering CRI-O factory
Oct 30 00:04:50.568949 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.565071    4441 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:04:50.568949 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.565128    4441 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:04:50.568949 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.565151    4441 factory.go:103] Registering Raw factory
Oct 30 00:04:50.568949 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.565167    4441 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:04:50.568949 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.567141    4441 manager.go:319] Starting recovery of all containers
Oct 30 00:04:50.568949 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.568874    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41e90ea1-6a7c-4007-89f4-bba585722b8a" volumeName="kubernetes.io/empty-dir/41e90ea1-6a7c-4007-89f4-bba585722b8a-tmp-dir" seLinuxMountContext=""
Oct 30 00:04:50.569334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.568942    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e052a3c-35d6-4e5a-bd1c-8fd962f8effe" volumeName="kubernetes.io/configmap/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-daemon-config" seLinuxMountContext=""
Oct 30 00:04:50.569334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569005    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e052a3c-35d6-4e5a-bd1c-8fd962f8effe" volumeName="kubernetes.io/projected/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-kube-api-access-rffhf" seLinuxMountContext=""
Oct 30 00:04:50.569334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569064    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23960d87-bf90-43c3-9ee6-464afa4c42ea" volumeName="kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:04:50.569334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569089    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23960d87-bf90-43c3-9ee6-464afa4c42ea" volumeName="kubernetes.io/projected/23960d87-bf90-43c3-9ee6-464afa4c42ea-kube-api-access-52b7g" seLinuxMountContext=""
Oct 30 00:04:50.569334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569119    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccef4fa8-30d6-4e12-b2f1-470b13955cc5" volumeName="kubernetes.io/projected/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-kube-api-access-gpklh" seLinuxMountContext=""
Oct 30 00:04:50.569334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569138    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23960d87-bf90-43c3-9ee6-464afa4c42ea" volumeName="kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 00:04:50.569334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569152    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3" volumeName="kubernetes.io/projected/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3-kube-api-access-xxq4w" seLinuxMountContext=""
Oct 30 00:04:50.569334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569205    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a055af25-0d4a-4b69-b395-259f12a4f363" volumeName="kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl" seLinuxMountContext=""
Oct 30 00:04:50.569334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569234    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d74677bd-f7f3-4a55-b5e7-0216819f5400" volumeName="kubernetes.io/secret/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 00:04:50.569334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569247    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23960d87-bf90-43c3-9ee6-464afa4c42ea" volumeName="kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 00:04:50.569334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569275    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="657fcf3d-0fff-4894-94dc-2dc1b84a9faf" volumeName="kubernetes.io/projected/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-api-access-8pbbk" seLinuxMountContext=""
Oct 30 00:04:50.569334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569301    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccef4fa8-30d6-4e12-b2f1-470b13955cc5" volumeName="kubernetes.io/empty-dir/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-tuned" seLinuxMountContext=""
Oct 30 00:04:50.569334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569317    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="854339cf-9de2-4d3f-9275-20cb99cb7c89" volumeName="kubernetes.io/projected/854339cf-9de2-4d3f-9275-20cb99cb7c89-kube-api-access-jv7zp" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569366    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d2ad09d-f117-46ba-bf91-9dbcccca5748" volumeName="kubernetes.io/projected/8d2ad09d-f117-46ba-bf91-9dbcccca5748-kube-api-access-2wtc6" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569417    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf392012-ba7c-4999-b0fe-f0ae7cee60f7" volumeName="kubernetes.io/secret/bf392012-ba7c-4999-b0fe-f0ae7cee60f7-agent-certs" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569431    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f2407f8-44bd-4a86-af6d-66429fb7ecfd" volumeName="kubernetes.io/projected/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-kube-api-access-bpx4v" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569474    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66a3d601-1733-40ac-834e-7378e3098542" volumeName="kubernetes.io/projected/66a3d601-1733-40ac-834e-7378e3098542-kube-api-access-nwjfl" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569494    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccef4fa8-30d6-4e12-b2f1-470b13955cc5" volumeName="kubernetes.io/empty-dir/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-tmp" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569506    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5" volumeName="kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569549    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="854339cf-9de2-4d3f-9275-20cb99cb7c89" volumeName="kubernetes.io/configmap/854339cf-9de2-4d3f-9275-20cb99cb7c89-iptables-alerter-script" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569562    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3" volumeName="kubernetes.io/secret/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3-cert" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569575    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d74677bd-f7f3-4a55-b5e7-0216819f5400" volumeName="kubernetes.io/projected/d74677bd-f7f3-4a55-b5e7-0216819f5400-kube-api-access-7xwzm" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569607    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f2407f8-44bd-4a86-af6d-66429fb7ecfd" volumeName="kubernetes.io/configmap/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-config-volume" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569619    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f2407f8-44bd-4a86-af6d-66429fb7ecfd" volumeName="kubernetes.io/empty-dir/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-tmp-dir" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569651    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="657fcf3d-0fff-4894-94dc-2dc1b84a9faf" volumeName="kubernetes.io/empty-dir/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-data-volume" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569676    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d74677bd-f7f3-4a55-b5e7-0216819f5400" volumeName="kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-env-overrides" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569689    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d74677bd-f7f3-4a55-b5e7-0216819f5400" volumeName="kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569731    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e052a3c-35d6-4e5a-bd1c-8fd962f8effe" volumeName="kubernetes.io/configmap/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569758    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f" volumeName="kubernetes.io/configmap/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-serviceca" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569772    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41e90ea1-6a7c-4007-89f4-bba585722b8a" volumeName="kubernetes.io/projected/41e90ea1-6a7c-4007-89f4-bba585722b8a-kube-api-access-8mtkj" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569784    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55ce16b2-ae85-4046-83b6-0fd872bca1a8" volumeName="kubernetes.io/projected/55ce16b2-ae85-4046-83b6-0fd872bca1a8-kube-api-access-4f9bp" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569826    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="657fcf3d-0fff-4894-94dc-2dc1b84a9faf" volumeName="kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569840    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf392012-ba7c-4999-b0fe-f0ae7cee60f7" volumeName="kubernetes.io/configmap/bf392012-ba7c-4999-b0fe-f0ae7cee60f7-konnectivity-ca" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569869    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d74677bd-f7f3-4a55-b5e7-0216819f5400" volumeName="kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovnkube-config" seLinuxMountContext=""
Oct 30 00:04:50.569919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569891    4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f" volumeName="kubernetes.io/projected/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-kube-api-access-hw6bf" seLinuxMountContext=""
Oct 30 00:04:50.570690 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569913    4441 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:04:50.570690 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.569955    4441 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:04:50.583720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.583695    4441 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:04:50.583900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.583880    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:04:50.584256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.584234    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:04:50.600933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.600909    4441 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:04:50.600933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.600933    4441 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:04:50.601122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.600954    4441 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:04:50.601122 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:50.601010    4441 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 00:04:50.609217 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.609180    4441 generic.go:356] "Generic (PLEG): container finished" podID="23960d87-bf90-43c3-9ee6-464afa4c42ea" containerID="5f2680238bd6cec7df87d7116835a1ce3b0660d58410d3e7612cccda256a677b" exitCode=0
Oct 30 00:04:50.609217 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.609217    4441 generic.go:356] "Generic (PLEG): container finished" podID="23960d87-bf90-43c3-9ee6-464afa4c42ea" containerID="2d00de2dc928ce79d59da7729c2a6896810a59639585ed7eee0b7701981d6628" exitCode=0
Oct 30 00:04:50.609351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.609226    4441 generic.go:356] "Generic (PLEG): container finished" podID="23960d87-bf90-43c3-9ee6-464afa4c42ea" containerID="3ecf0d5e69bc467e97200bc2687c98d902f9d3e03f42f1cf82ce142fc816051a" exitCode=0
Oct 30 00:04:50.609351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.609233    4441 generic.go:356] "Generic (PLEG): container finished" podID="23960d87-bf90-43c3-9ee6-464afa4c42ea" containerID="971c3512a44a0f0619254b977f8bd02384582868883a5160994c398bed82370a" exitCode=0
Oct 30 00:04:50.609351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.609240    4441 generic.go:356] "Generic (PLEG): container finished" podID="23960d87-bf90-43c3-9ee6-464afa4c42ea" containerID="8aedfc13ee033075736ecf300dc56ccb41340847e4f16a20256f384a5f972eac" exitCode=0
Oct 30 00:04:50.609351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.609248    4441 generic.go:356] "Generic (PLEG): container finished" podID="23960d87-bf90-43c3-9ee6-464afa4c42ea" containerID="6803578ebd7b12ede7330bd358a899512d064e3c309417775a5139be0df8d95e" exitCode=0
Oct 30 00:04:50.614386 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.614362    4441 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:04:50.614520 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.614477    4441 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 00:04:50.615544 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.615525    4441 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:04:50.615942 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.615926    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:04:50.616261 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.616237    4441 generic.go:356] "Generic (PLEG): container finished" podID="d74677bd-f7f3-4a55-b5e7-0216819f5400" containerID="742b573dce28a53b4c5075bb2ab55d8e9b84860536ed704eabf75eee73dc5b36" exitCode=1
Oct 30 00:04:50.627233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.627209    4441 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:04:50.627707 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.627688    4441 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:04:50.634653 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.634629    4441 generic.go:356] "Generic (PLEG): container finished" podID="689ea6de40d51a70fe308b471157700c" containerID="d17c2fb640d1cce2b5bcd574a50285189357d76a1834555438b7ba5040d0ced2" exitCode=0
Oct 30 00:04:50.634735 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.634655    4441 manager.go:324] Recovery completed
Oct 30 00:04:50.647566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.647549    4441 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:04:50.647566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.647562    4441 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:04:50.647695 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.647578    4441 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:04:50.647695 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.647678    4441 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 00:04:50.647695 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.647688    4441 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 00:04:50.647807 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.647702    4441 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 00:04:50.647807 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.647707    4441 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 00:04:50.647807 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.647712    4441 policy_none.go:50] "Start"
Oct 30 00:04:50.647807 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.647721    4441 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:04:50.647807 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.647729    4441 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:04:50.647964 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.647841    4441 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 00:04:50.647964 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.647848    4441 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 00:04:50.647964 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.647853    4441 policy_none.go:44] "Start"
Oct 30 00:04:50.651539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.651511    4441 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:04:50.654071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.654058    4441 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:04:50.654701 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:50.654087    4441 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:04:50.654750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.654707    4441 server.go:87] "Starting device plugin registration server"
Oct 30 00:04:50.655045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.655018    4441 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:04:50.655089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.655048    4441 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:04:50.655160 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.655142    4441 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:04:50.655283 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.655247    4441 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:04:50.655343 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.655292    4441 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:04:50.655343 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.655299    4441 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:04:50.655421 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.655331    4441 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:04:50.655281615Z","UUID":"9095cc47-75ca-409b-975c-98a1ce0ceb9e","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:04:50.657165 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.657146    4441 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:04:50.657260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.657176    4441 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:04:50.657260 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:50.657147    4441 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:04:50.701870 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.701814    4441 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"]
Oct 30 00:04:50.702319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.702290    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal" event={"ID":"e3314e96d9e96380a527867d39d2fde5","Type":"ContainerStarted","Data":"cac5ad08c9a57b7051ca6abf9de2ecb4ded9fd974b32663dcab1b079a65e3e4c"}
Oct 30 00:04:50.702420 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.702326    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal" event={"ID":"e3314e96d9e96380a527867d39d2fde5","Type":"ContainerStarted","Data":"59edc93c744b47b3b583bf3add9433bad217423f019c2e57e96c6a8c467214af"}
Oct 30 00:04:50.702420 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.702339    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal" event={"ID":"689ea6de40d51a70fe308b471157700c","Type":"ContainerStarted","Data":"c2e2c091413db34aefbfbf9152575fc056a8c5b874932db9cda447e2253280d4"}
Oct 30 00:04:50.702420 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.702345    4441 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.702575 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.702351    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal" event={"ID":"689ea6de40d51a70fe308b471157700c","Type":"ContainerDied","Data":"d17c2fb640d1cce2b5bcd574a50285189357d76a1834555438b7ba5040d0ced2"}
Oct 30 00:04:50.702575 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.702458    4441 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.702575 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.702469    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal" event={"ID":"689ea6de40d51a70fe308b471157700c","Type":"ContainerStarted","Data":"f72d6d823f592be1cab0e2a8af30b7bf158bc2a39b11875d74a6a44a0dbb1ae6"}
Oct 30 00:04:50.724988 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.724970    4441 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:04:50.725072 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:50.725004    4441 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-69.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.725522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.725509    4441 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:04:50.725557 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:50.725539    4441 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.755648 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.755633    4441 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:04:50.756801 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.756787    4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:04:50.756873 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.756813    4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:04:50.756873 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.756829    4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:04:50.756873 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.756859    4441 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.789010 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.788994    4441 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.789067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.789055    4441 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.815096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.815075    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/689ea6de40d51a70fe308b471157700c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" (UID: \"689ea6de40d51a70fe308b471157700c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.815180 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.815100    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/689ea6de40d51a70fe308b471157700c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" (UID: \"689ea6de40d51a70fe308b471157700c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.815180 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.815122    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3314e96d9e96380a527867d39d2fde5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.ec2.internal\" (UID: \"e3314e96d9e96380a527867d39d2fde5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.915530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.915508    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3314e96d9e96380a527867d39d2fde5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.ec2.internal\" (UID: \"e3314e96d9e96380a527867d39d2fde5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.915599 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.915542    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3314e96d9e96380a527867d39d2fde5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.ec2.internal\" (UID: \"e3314e96d9e96380a527867d39d2fde5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.915599 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.915549    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/689ea6de40d51a70fe308b471157700c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" (UID: \"689ea6de40d51a70fe308b471157700c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.915599 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.915574    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/689ea6de40d51a70fe308b471157700c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" (UID: \"689ea6de40d51a70fe308b471157700c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.915681 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.915602    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/689ea6de40d51a70fe308b471157700c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" (UID: \"689ea6de40d51a70fe308b471157700c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:50.915681 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:50.915623    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/689ea6de40d51a70fe308b471157700c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" (UID: \"689ea6de40d51a70fe308b471157700c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:51.554007 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.553973    4441 apiserver.go:52] "Watching apiserver"
Oct 30 00:04:51.592352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.592323    4441 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:04:51.592762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.592745    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-rrphh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal","openshift-multus/network-metrics-daemon-6cnnk","openshift-multus/multus-additional-cni-plugins-8k6ms","openshift-network-operator/iptables-alerter-hnp8z","kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal","openshift-dns/dns-default-xzfnp","openshift-ingress-canary/ingress-canary-qxjrt","openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v","openshift-cluster-node-tuning-operator/tuned-wtbxs","openshift-insights/insights-runtime-extractor-xvj9n","kube-system/konnectivity-agent-kmg9c","openshift-image-registry/node-ca-6hq6g","kube-system/global-pull-secret-syncer-v5tjs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k","openshift-network-diagnostics/network-check-target-tr4w8","openshift-dns/node-resolver-jvrpq","openshift-ovn-kubernetes/ovnkube-node-sf6zz"]
Oct 30 00:04:51.593635 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.593532    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:51.599931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.599911    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:51.600648 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.600630    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:51.608280 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.608255    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:04:51.608280 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.608270    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:04:51.608412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.608295    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k8n8m\""
Oct 30 00:04:51.608412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.608332    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m7wf2\""
Oct 30 00:04:51.608522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.608440    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:04:51.608522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.608500    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wvd28\""
Oct 30 00:04:51.608522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.608516    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n6hlg\""
Oct 30 00:04:51.608621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.608525    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thx59\""
Oct 30 00:04:51.608808 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.608794    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4xzk4\""
Oct 30 00:04:51.608862 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.608813    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:04:51.608862 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.608825    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:04:51.608971 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.608956    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:04:51.609021 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.608963    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:04:51.609021 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.609017    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:04:51.609139 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.609053    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:04:51.609208 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.609176    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:04:51.609260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.609218    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:04:51.609260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.609229    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:04:51.609260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.609222    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:04:51.609371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.609276    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:04:51.609371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.609280    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:04:51.609371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.609287    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:04:51.609371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.609280    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:04:51.609371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.609303    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:04:51.609371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.609341    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:04:51.620020 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.619973    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:04:51.620020 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.619974    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:04:51.620121 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.619992    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:04:51.620121 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.619999    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:04:51.620363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.620337    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:04:51.620363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.620357    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:04:51.620781 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.620765    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:04:51.620858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.620806    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:04:51.620858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.620816    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 00:04:51.620858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.620812    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:04:51.620858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.620835    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:04:51.620858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.620834    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:04:51.620858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.620857    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 00:04:51.621125 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.620928    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:04:51.621125 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.620934    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:04:51.621125 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621040    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:04:51.621125 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621084    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:04:51.621263 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621152    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:04:51.621298 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621275    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:04:51.621480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621467    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:04:51.621480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621474    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:04:51.621551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621482    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:04:51.621551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621528    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-pxfs7\""
Oct 30 00:04:51.621551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621534    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ntcgl\""
Oct 30 00:04:51.621649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621552    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqlwc\""
Oct 30 00:04:51.621649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621580    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r8lkn\""
Oct 30 00:04:51.621649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621634    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:04:51.621742 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621680    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9vr82\""
Oct 30 00:04:51.621742 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621715    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h92x8\""
Oct 30 00:04:51.621808 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621776    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-l5flb\""
Oct 30 00:04:51.621808 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621785    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fmnxz\""
Oct 30 00:04:51.621881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621832    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kct9w\""
Oct 30 00:04:51.621912 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.621886    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n7fsr\""
Oct 30 00:04:51.638441 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.638422    4441 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:51.638584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.638571    4441 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:51.658697 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.658680    4441 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:04:51.658763 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:51.658714    4441 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-69.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:51.659259 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.659246    4441 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:04:51.659293 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:51.659275    4441 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.ec2.internal"
Oct 30 00:04:51.664195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.664182    4441 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:04:51.688441 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.688405    4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmg9c_bf392012-ba7c-4999-b0fe-f0ae7cee60f7/konnectivity-agent/0.log" limit=1024
Oct 30 00:04:51.720151 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720123    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-registration-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.720281 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720155    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-cnibin\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.720281 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720188    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-multus-certs\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.720281 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720253    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-sys-fs\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.720409 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720295    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-serviceca\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:51.720442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720430    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.720491 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720468    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:51.720545 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720514    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-var-lib-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.720578 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720554    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-serviceca\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:51.720615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720580    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysctl-d\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.720652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720625    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.720698 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720673    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-config-volume\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:51.720754 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720735    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.720823 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720806    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:51.720870 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720828    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-config-volume\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:51.720870 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720839    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.720870 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720867    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-cni-bin\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.721015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720897    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-cni-netd\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.721015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720936    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/854339cf-9de2-4d3f-9275-20cb99cb7c89-iptables-alerter-script\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:51.721015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.720978    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-modprobe-d\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.721156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721055    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-socket-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.721156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721077    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-crio-socket\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:51.721156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721111    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-cni-bin\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.721156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721132    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovn-node-metrics-cert\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.721156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721147    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-lib-modules\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.721324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721170    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/854339cf-9de2-4d3f-9275-20cb99cb7c89-iptables-alerter-script\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:51.721324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721188    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw6bf\" (UniqueName: \"kubernetes.io/projected/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-kube-api-access-hw6bf\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:51.721324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721219    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-daemon-config\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.721324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721299    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovn-node-metrics-cert\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.721324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721313    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-systemd-units\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.721551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721372    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-env-overrides\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.721551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721439    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdwl\" (UniqueName: \"kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl\") pod \"network-check-target-tr4w8\" (UID: \"a055af25-0d4a-4b69-b395-259f12a4f363\") " pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:51.721551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721494    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wtc6\" (UniqueName: \"kubernetes.io/projected/8d2ad09d-f117-46ba-bf91-9dbcccca5748-kube-api-access-2wtc6\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.721551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721524    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-host\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:51.721551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721501    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-daemon-config\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.721551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721542    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:51.721802 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721566    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:51.721802 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721584    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-node-log\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.721802 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721607    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-tuned\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.721802 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721646    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3-cert\") pod \"ingress-canary-qxjrt\" (UID: \"9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3\") " pod="openshift-ingress-canary/ingress-canary-qxjrt"
Oct 30 00:04:51.721802 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721673    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-tuned\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.721802 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721709    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-env-overrides\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.721802 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721724    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:51.721802 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721711    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xwzm\" (UniqueName: \"kubernetes.io/projected/d74677bd-f7f3-4a55-b5e7-0216819f5400-kube-api-access-7xwzm\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.722278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721822    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:51.722278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721859    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-log-socket\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.722278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721872    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3-cert\") pod \"ingress-canary-qxjrt\" (UID: \"9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3\") " pod="openshift-ingress-canary/ingress-canary-qxjrt"
Oct 30 00:04:51.722278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721962    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysctl-conf\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.722278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.721994    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-run\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.722278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722064    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-original-pull-secret\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:51.722278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722074    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpx4v\" (UniqueName: \"kubernetes.io/projected/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-kube-api-access-bpx4v\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:51.722278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722109    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pbbk\" (UniqueName: \"kubernetes.io/projected/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-api-access-8pbbk\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:51.722278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722209    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.722278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722227    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-run-netns\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.722278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722242    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-run-ovn-kubernetes\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.722278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722279    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpklh\" (UniqueName: \"kubernetes.io/projected/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-kube-api-access-gpklh\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722295    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-etc-kubernetes\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722310    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovnkube-script-lib\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722341    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-kubernetes\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722415    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722447    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-tmp-dir\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722485    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovnkube-script-lib\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722498    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-tmp-dir\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722502    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwjfl\" (UniqueName: \"kubernetes.io/projected/66a3d601-1733-40ac-834e-7378e3098542-kube-api-access-nwjfl\") pod \"migrator-66df5d496b-4jj6v\" (UID: \"66a3d601-1733-40ac-834e-7378e3098542\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722540    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722567    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-k8s-cni-cncf-io\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722604    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rffhf\" (UniqueName: \"kubernetes.io/projected/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-kube-api-access-rffhf\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722625    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-ovn\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722641    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722658    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-sys\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722696    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-host\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722726    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-device-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722747    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52b7g\" (UniqueName: \"kubernetes.io/projected/23960d87-bf90-43c3-9ee6-464afa4c42ea-kube-api-access-52b7g\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722763    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-cnibin\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722783    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e90ea1-6a7c-4007-89f4-bba585722b8a-tmp-dir\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722798    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-etc-selinux\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.722816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722816    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bf392012-ba7c-4999-b0fe-f0ae7cee60f7-konnectivity-ca\") pod \"konnectivity-agent-kmg9c\" (UID: \"bf392012-ba7c-4999-b0fe-f0ae7cee60f7\") " pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722872    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722891    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e90ea1-6a7c-4007-89f4-bba585722b8a-tmp-dir\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722905    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-netns\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.722996    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-slash\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723009    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bf392012-ba7c-4999-b0fe-f0ae7cee60f7-konnectivity-ca\") pod \"konnectivity-agent-kmg9c\" (UID: \"bf392012-ba7c-4999-b0fe-f0ae7cee60f7\") " pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723034    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7zp\" (UniqueName: \"kubernetes.io/projected/854339cf-9de2-4d3f-9275-20cb99cb7c89-kube-api-access-jv7zp\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723039    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23960d87-bf90-43c3-9ee6-464afa4c42ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723064    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtkj\" (UniqueName: \"kubernetes.io/projected/41e90ea1-6a7c-4007-89f4-bba585722b8a-kube-api-access-8mtkj\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723082    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bf392012-ba7c-4999-b0fe-f0ae7cee60f7-agent-certs\") pod \"konnectivity-agent-kmg9c\" (UID: \"bf392012-ba7c-4999-b0fe-f0ae7cee60f7\") " pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723099    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxq4w\" (UniqueName: \"kubernetes.io/projected/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3-kube-api-access-xxq4w\") pod \"ingress-canary-qxjrt\" (UID: \"9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3\") " pod="openshift-ingress-canary/ingress-canary-qxjrt"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723122    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-cni-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723146    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-kubelet-config\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723168    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-kubelet\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723223    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-systemd\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723279    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-data-volume\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723308    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bf392012-ba7c-4999-b0fe-f0ae7cee60f7-agent-certs\") pod \"konnectivity-agent-kmg9c\" (UID: \"bf392012-ba7c-4999-b0fe-f0ae7cee60f7\") " pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723313    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-conf-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723362    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-dbus\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723407    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41e90ea1-6a7c-4007-89f4-bba585722b8a-hosts-file\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723445    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-var-lib-kubelet\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723445    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-data-volume\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723478    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovnkube-config\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723509    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/854339cf-9de2-4d3f-9275-20cb99cb7c89-host-slash\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723536    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-system-cni-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723565    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-cni-binary-copy\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723615    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-kubelet\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.723641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723646    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d74677bd-f7f3-4a55-b5e7-0216819f5400-ovnkube-config\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.724518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723691    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-hostroot\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.724518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723747    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-cni-binary-copy\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.724518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723751    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-etc-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.724518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723781    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-system-cni-dir\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.724518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723802    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-socket-dir-parent\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.724518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723817    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-cni-multus\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.724518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723873    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f9bp\" (UniqueName: \"kubernetes.io/projected/55ce16b2-ae85-4046-83b6-0fd872bca1a8-kube-api-access-4f9bp\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:51.724518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723903    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysconfig\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.724518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723928    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-systemd\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.724518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.723953    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-tmp\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.724518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.724010    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-tmp\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.741826 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.741805    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdwl\" (UniqueName: \"kubernetes.io/projected/a055af25-0d4a-4b69-b395-259f12a4f363-kube-api-access-ltdwl\") pod \"network-check-target-tr4w8\" (UID: \"a055af25-0d4a-4b69-b395-259f12a4f363\") " pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:51.742015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.741996    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xwzm\" (UniqueName: \"kubernetes.io/projected/d74677bd-f7f3-4a55-b5e7-0216819f5400-kube-api-access-7xwzm\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.742015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.742007    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wtc6\" (UniqueName: \"kubernetes.io/projected/8d2ad09d-f117-46ba-bf91-9dbcccca5748-kube-api-access-2wtc6\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.742131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.742069    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpx4v\" (UniqueName: \"kubernetes.io/projected/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-kube-api-access-bpx4v\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:51.742131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.742069    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw6bf\" (UniqueName: \"kubernetes.io/projected/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-kube-api-access-hw6bf\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:51.747660 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.747626    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpklh\" (UniqueName: \"kubernetes.io/projected/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-kube-api-access-gpklh\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.747931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.747910    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxq4w\" (UniqueName: \"kubernetes.io/projected/9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3-kube-api-access-xxq4w\") pod \"ingress-canary-qxjrt\" (UID: \"9fb4d953-9d64-45fb-bbff-2d32e7cbdaf3\") " pod="openshift-ingress-canary/ingress-canary-qxjrt"
Oct 30 00:04:51.748149 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.748128    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f9bp\" (UniqueName: \"kubernetes.io/projected/55ce16b2-ae85-4046-83b6-0fd872bca1a8-kube-api-access-4f9bp\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:51.748255 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.748225    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rffhf\" (UniqueName: \"kubernetes.io/projected/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-kube-api-access-rffhf\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.748363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.748226    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtkj\" (UniqueName: \"kubernetes.io/projected/41e90ea1-6a7c-4007-89f4-bba585722b8a-kube-api-access-8mtkj\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:51.748363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.748271    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52b7g\" (UniqueName: \"kubernetes.io/projected/23960d87-bf90-43c3-9ee6-464afa4c42ea-kube-api-access-52b7g\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.748363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.748271    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwjfl\" (UniqueName: \"kubernetes.io/projected/66a3d601-1733-40ac-834e-7378e3098542-kube-api-access-nwjfl\") pod \"migrator-66df5d496b-4jj6v\" (UID: \"66a3d601-1733-40ac-834e-7378e3098542\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jj6v"
Oct 30 00:04:51.748363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.748228    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pbbk\" (UniqueName: \"kubernetes.io/projected/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-kube-api-access-8pbbk\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:51.748490 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.748385    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7zp\" (UniqueName: \"kubernetes.io/projected/854339cf-9de2-4d3f-9275-20cb99cb7c89-kube-api-access-jv7zp\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:51.824480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824388    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-multus-certs\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.824480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824429    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-sys-fs\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.824480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824443    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-multus-certs\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.824480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824453    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:51.824480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824474    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-var-lib-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824495    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysctl-d\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824541    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-var-lib-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824547    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824586    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824598    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824586    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-sys-fs\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824614    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824645    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824684    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-cni-bin\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824684    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysctl-d\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824714    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-cni-netd\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824743    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-cni-bin\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824738    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-modprobe-d\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:51.824692    4441 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824757    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-cni-netd\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824768    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-socket-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824785    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-crio-socket\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824809    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-cni-bin\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824825    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-socket-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:51.824697    4441 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:51.824855    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs podName:55ce16b2-ae85-4046-83b6-0fd872bca1a8 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:52.324834937 +0000 UTC m=+1.850128811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs") pod "network-metrics-daemon-6cnnk" (UID: "55ce16b2-ae85-4046-83b6-0fd872bca1a8") : secret "metrics-daemon-secret" not found
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824868    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-modprobe-d\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824877    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-lib-modules\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824873    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-cni-bin\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824874    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-crio-socket\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:51.824893 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:51.824906    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls podName:3f2407f8-44bd-4a86-af6d-66429fb7ecfd nodeName:}" failed. No retries permitted until 2026-10-30 00:04:52.324887735 +0000 UTC m=+1.850181594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls") pod "dns-default-xzfnp" (UID: "3f2407f8-44bd-4a86-af6d-66429fb7ecfd") : secret "dns-default-metrics-tls" not found
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824942    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-systemd-units\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824955    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-lib-modules\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824971    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-host\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.824997    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825004    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-systemd-units\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825046    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-node-log\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825075    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-log-socket\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:51.825099    4441 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825143    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f-host\") pod \"node-ca-6hq6g\" (UID: \"40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f\") " pod="openshift-image-registry/node-ca-6hq6g"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:51.825154    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls podName:657fcf3d-0fff-4894-94dc-2dc1b84a9faf nodeName:}" failed. No retries permitted until 2026-10-30 00:04:52.325139659 +0000 UTC m=+1.850433518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvj9n" (UID: "657fcf3d-0fff-4894-94dc-2dc1b84a9faf") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825097    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysctl-conf\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825185    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-run\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825204    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-run-netns\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825141    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-log-socket\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825143    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-node-log\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825216    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysctl-conf\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825239    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-run\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825251    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-run-netns\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825239    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-run-ovn-kubernetes\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825262    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-run-ovn-kubernetes\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825280    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-etc-kubernetes\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825296    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-kubernetes\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825317    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825342    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-k8s-cni-cncf-io\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825347    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-etc-kubernetes\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825366    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-ovn\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825375    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-kubernetes\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825387    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-k8s-cni-cncf-io\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825391    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825401    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-ovn\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825403    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825414    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-sys\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825429    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-host\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825434    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825443    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-device-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825463    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-sys\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825481    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-device-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825480    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-host\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825494    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-cnibin\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825511    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-etc-selinux\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825527    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-netns\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825546    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-slash\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825558    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-cnibin\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825575    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-cni-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825581    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-etc-selinux\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825584    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-run-netns\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825598    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-slash\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825600    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-kubelet-config\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825626    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-kubelet\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825643    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-cni-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825648    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-systemd\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825664    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-kubelet-config\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825670    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-conf-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825693    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-dbus\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825702    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-run-systemd\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825705    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-conf-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825708    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41e90ea1-6a7c-4007-89f4-bba585722b8a-hosts-file\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825741    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-var-lib-kubelet\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825752    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41e90ea1-6a7c-4007-89f4-bba585722b8a-hosts-file\") pod \"node-resolver-jvrpq\" (UID: \"41e90ea1-6a7c-4007-89f4-bba585722b8a\") " pod="openshift-dns/node-resolver-jvrpq"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825677    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-host-kubelet\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825771    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5-dbus\") pod \"global-pull-secret-syncer-v5tjs\" (UID: \"3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5\") " pod="kube-system/global-pull-secret-syncer-v5tjs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825798    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/854339cf-9de2-4d3f-9275-20cb99cb7c89-host-slash\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825817    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-var-lib-kubelet\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825764    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/854339cf-9de2-4d3f-9275-20cb99cb7c89-host-slash\") pod \"iptables-alerter-hnp8z\" (UID: \"854339cf-9de2-4d3f-9275-20cb99cb7c89\") " pod="openshift-network-operator/iptables-alerter-hnp8z"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825862    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-system-cni-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825887    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-kubelet\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825911    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-hostroot\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825950    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-system-cni-dir\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825950    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-hostroot\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825952    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-kubelet\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825969    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-etc-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825990    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-system-cni-dir\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.825996    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d74677bd-f7f3-4a55-b5e7-0216819f5400-etc-openvswitch\") pod \"ovnkube-node-sf6zz\" (UID: \"d74677bd-f7f3-4a55-b5e7-0216819f5400\") " pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.826005    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-socket-dir-parent\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.826040    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-cni-multus\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.826052    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-system-cni-dir\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.825978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.826057    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysconfig\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.828039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.826082    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-systemd\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.828039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.826098    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-host-var-lib-cni-multus\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.828039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.826102    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e052a3c-35d6-4e5a-bd1c-8fd962f8effe-multus-socket-dir-parent\") pod \"multus-rrphh\" (UID: \"0e052a3c-35d6-4e5a-bd1c-8fd962f8effe\") " pod="openshift-multus/multus-rrphh"
Oct 30 00:04:51.828039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.826107    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-registration-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.828039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.826136    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-cnibin\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.828039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.826145    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-sysconfig\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.828039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.826152    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d2ad09d-f117-46ba-bf91-9dbcccca5748-registration-dir\") pod \"aws-ebs-csi-driver-node-hrj5k\" (UID: \"8d2ad09d-f117-46ba-bf91-9dbcccca5748\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrj5k"
Oct 30 00:04:51.828039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.826193    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccef4fa8-30d6-4e12-b2f1-470b13955cc5-etc-systemd\") pod \"tuned-wtbxs\" (UID: \"ccef4fa8-30d6-4e12-b2f1-470b13955cc5\") " pod="openshift-cluster-node-tuning-operator/tuned-wtbxs"
Oct 30 00:04:51.828039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.826206    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23960d87-bf90-43c3-9ee6-464afa4c42ea-cnibin\") pod \"multus-additional-cni-plugins-8k6ms\" (UID: \"23960d87-bf90-43c3-9ee6-464afa4c42ea\") " pod="openshift-multus/multus-additional-cni-plugins-8k6ms"
Oct 30 00:04:51.871424 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:51.871398    4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmg9c_bf392012-ba7c-4999-b0fe-f0ae7cee60f7/konnectivity-agent/0.log" limit=1024
Oct 30 00:04:52.329738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:52.329705    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:52.329738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:52.329741    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:52.329928 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:52.329776    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:52.329928 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:52.329857    4441 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:04:52.329928 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:52.329911    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs podName:55ce16b2-ae85-4046-83b6-0fd872bca1a8 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:53.32989922 +0000 UTC m=+2.855193080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs") pod "network-metrics-daemon-6cnnk" (UID: "55ce16b2-ae85-4046-83b6-0fd872bca1a8") : secret "metrics-daemon-secret" not found
Oct 30 00:04:52.330042 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:52.329857    4441 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:52.330042 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:52.329997    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls podName:657fcf3d-0fff-4894-94dc-2dc1b84a9faf nodeName:}" failed. No retries permitted until 2026-10-30 00:04:53.329984767 +0000 UTC m=+2.855278621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvj9n" (UID: "657fcf3d-0fff-4894-94dc-2dc1b84a9faf") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:52.330042 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:52.329857    4441 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:04:52.330130 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:52.330043    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls podName:3f2407f8-44bd-4a86-af6d-66429fb7ecfd nodeName:}" failed. No retries permitted until 2026-10-30 00:04:53.330019313 +0000 UTC m=+2.855313168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls") pod "dns-default-xzfnp" (UID: "3f2407f8-44bd-4a86-af6d-66429fb7ecfd") : secret "dns-default-metrics-tls" not found
Oct 30 00:04:53.060416 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:53.060384    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:53.338491 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:53.338407    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:53.338491 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:53.338441    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:53.338491 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:53.338466    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:53.338721 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:53.338552    4441 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:04:53.338721 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:53.338608    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs podName:55ce16b2-ae85-4046-83b6-0fd872bca1a8 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:55.338594696 +0000 UTC m=+4.863888551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs") pod "network-metrics-daemon-6cnnk" (UID: "55ce16b2-ae85-4046-83b6-0fd872bca1a8") : secret "metrics-daemon-secret" not found
Oct 30 00:04:53.338721 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:53.338558    4441 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:53.338721 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:53.338679    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls podName:657fcf3d-0fff-4894-94dc-2dc1b84a9faf nodeName:}" failed. No retries permitted until 2026-10-30 00:04:55.338665357 +0000 UTC m=+4.863959211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvj9n" (UID: "657fcf3d-0fff-4894-94dc-2dc1b84a9faf") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:53.338852 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:53.338787    4441 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:04:53.340891 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:53.340871    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2407f8-44bd-4a86-af6d-66429fb7ecfd-metrics-tls\") pod \"dns-default-xzfnp\" (UID: \"3f2407f8-44bd-4a86-af6d-66429fb7ecfd\") " pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:53.401283 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:53.401258    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:53.726436 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:53.726283    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xzfnp"]
Oct 30 00:04:53.730046 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:53.729997    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2407f8_44bd_4a86_af6d_66429fb7ecfd.slice/crio-023421000acbf28f8957cd98f073ab89b5359b01140b391179af7737379c5a71 WatchSource:0}: Error finding container 023421000acbf28f8957cd98f073ab89b5359b01140b391179af7737379c5a71: Status 404 returned error can't find the container with id 023421000acbf28f8957cd98f073ab89b5359b01140b391179af7737379c5a71
Oct 30 00:04:53.732020 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:53.731999    4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:04:53.877580 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:53.877562    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:04:53.877850 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:53.877834    4441 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:04:54.646632 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:54.646595    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xzfnp" event={"ID":"3f2407f8-44bd-4a86-af6d-66429fb7ecfd","Type":"ContainerStarted","Data":"023421000acbf28f8957cd98f073ab89b5359b01140b391179af7737379c5a71"}
Oct 30 00:04:55.353347 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:55.353326    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:55.353448 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:55.353363    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:55.353502 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:55.353459    4441 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:55.353538 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:04:55.353502    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls podName:657fcf3d-0fff-4894-94dc-2dc1b84a9faf nodeName:}" failed. No retries permitted until 2026-10-30 00:04:59.353491562 +0000 UTC m=+8.878785429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvj9n" (UID: "657fcf3d-0fff-4894-94dc-2dc1b84a9faf") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:04:55.355450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:55.355423    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ce16b2-ae85-4046-83b6-0fd872bca1a8-metrics-certs\") pod \"network-metrics-daemon-6cnnk\" (UID: \"55ce16b2-ae85-4046-83b6-0fd872bca1a8\") " pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:55.494721 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:55.494698    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6cnnk"
Oct 30 00:04:55.650750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:55.650719    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xzfnp" event={"ID":"3f2407f8-44bd-4a86-af6d-66429fb7ecfd","Type":"ContainerStarted","Data":"26ed86a9f58a61fa661a069f85f87cda9c8c899ea88810c03c78bc57b52a2953"}
Oct 30 00:04:55.651228 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:55.650777    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xzfnp" event={"ID":"3f2407f8-44bd-4a86-af6d-66429fb7ecfd","Type":"ContainerStarted","Data":"550d91ffc2a6ad5921797e91dcf7950d37b80b6b2b4f25468f74b99a297937fb"}
Oct 30 00:04:55.651228 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:55.650862    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:04:55.653143 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:55.653128    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:04:55.782792 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:55.782717    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6cnnk"]
Oct 30 00:04:55.785906 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:55.785880    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ce16b2_ae85_4046_83b6_0fd872bca1a8.slice/crio-6e9127b88b7fd591a1b1aa1fa11e4d545c79249399f098140d25f8e6c7de7ade WatchSource:0}: Error finding container 6e9127b88b7fd591a1b1aa1fa11e4d545c79249399f098140d25f8e6c7de7ade: Status 404 returned error can't find the container with id 6e9127b88b7fd591a1b1aa1fa11e4d545c79249399f098140d25f8e6c7de7ade
Oct 30 00:04:56.656518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:56.656488    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6cnnk" event={"ID":"55ce16b2-ae85-4046-83b6-0fd872bca1a8","Type":"ContainerStarted","Data":"6e9127b88b7fd591a1b1aa1fa11e4d545c79249399f098140d25f8e6c7de7ade"}
Oct 30 00:04:57.661437 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:57.661400    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6cnnk" event={"ID":"55ce16b2-ae85-4046-83b6-0fd872bca1a8","Type":"ContainerStarted","Data":"153a0103c8dd6e36005af7cf34ef99a48032ebe81297897a6b075e400ae4af05"}
Oct 30 00:04:57.661838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:57.661478    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6cnnk" event={"ID":"55ce16b2-ae85-4046-83b6-0fd872bca1a8","Type":"ContainerStarted","Data":"35be021859178c9d5fc8c74a3d53b1f6a1965e12e22c946f04990a2130d91335"}
Oct 30 00:04:59.378219 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:59.378175    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:59.380323 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:59.380300    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/657fcf3d-0fff-4894-94dc-2dc1b84a9faf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvj9n\" (UID: \"657fcf3d-0fff-4894-94dc-2dc1b84a9faf\") " pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:59.400245 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:59.400223    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xvj9n"
Oct 30 00:04:59.703250 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:59.703179    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xvj9n"]
Oct 30 00:04:59.706214 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:04:59.706186    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657fcf3d_0fff_4894_94dc_2dc1b84a9faf.slice/crio-a9444de84435c756b514f943b1d51139ba940533d9772a00a7e9830d9348cf9b WatchSource:0}: Error finding container a9444de84435c756b514f943b1d51139ba940533d9772a00a7e9830d9348cf9b: Status 404 returned error can't find the container with id a9444de84435c756b514f943b1d51139ba940533d9772a00a7e9830d9348cf9b
Oct 30 00:04:59.962891 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:04:59.962819    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sf6zz"
Oct 30 00:05:00.648614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:00.648535    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kmg9c"
Oct 30 00:05:00.671898 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:00.671868    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvj9n" event={"ID":"657fcf3d-0fff-4894-94dc-2dc1b84a9faf","Type":"ContainerStarted","Data":"26e488c92a738a14d55c56d0a463cbaa133e70151c0de28d73c836a25fdfa6cd"}
Oct 30 00:05:00.672013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:00.671915    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvj9n" event={"ID":"657fcf3d-0fff-4894-94dc-2dc1b84a9faf","Type":"ContainerStarted","Data":"81373d749f13867ac5e35da55239bb8f42def64c7e77b4a37f41185aaccd69c2"}
Oct 30 00:05:00.672013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:00.671935    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvj9n" event={"ID":"657fcf3d-0fff-4894-94dc-2dc1b84a9faf","Type":"ContainerStarted","Data":"a9444de84435c756b514f943b1d51139ba940533d9772a00a7e9830d9348cf9b"}
Oct 30 00:05:01.676672 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:01.676644    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvj9n" event={"ID":"657fcf3d-0fff-4894-94dc-2dc1b84a9faf","Type":"ContainerStarted","Data":"c195643652ad72934ca4ba0be9893d27cb56c91dc4f52df28f283ce487e56a6f"}
Oct 30 00:05:02.940927 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:02.940893    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76c8cdc68c-8jdsl"]
Oct 30 00:05:02.943135 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:02.943118    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:02.994191 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:02.994152    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 00:05:02.994363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:02.994206    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 00:05:02.994363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:02.994261    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 00:05:02.994363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:02.994276    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 00:05:02.994495 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:02.994480    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-smkfn\""
Oct 30 00:05:02.994886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:02.994869    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76c8cdc68c-8jdsl"]
Oct 30 00:05:03.102527 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.102496    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df9d6656-9a1a-445e-9b70-f3a9fad5038c-registry-certificates\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.102694 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.102540    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df9d6656-9a1a-445e-9b70-f3a9fad5038c-installation-pull-secrets\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.102694 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.102576    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df9d6656-9a1a-445e-9b70-f3a9fad5038c-image-registry-private-configuration\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.102694 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.102633    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df9d6656-9a1a-445e-9b70-f3a9fad5038c-trusted-ca\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.102694 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.102670    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df9d6656-9a1a-445e-9b70-f3a9fad5038c-ca-trust-extracted\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.102857 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.102737    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df9d6656-9a1a-445e-9b70-f3a9fad5038c-bound-sa-token\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.102857 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.102774    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z6ps\" (UniqueName: \"kubernetes.io/projected/df9d6656-9a1a-445e-9b70-f3a9fad5038c-kube-api-access-8z6ps\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.102857 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.102837    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df9d6656-9a1a-445e-9b70-f3a9fad5038c-registry-tls\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.204062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.203958    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df9d6656-9a1a-445e-9b70-f3a9fad5038c-registry-certificates\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.204062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.203999    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df9d6656-9a1a-445e-9b70-f3a9fad5038c-installation-pull-secrets\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.204062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.204019    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df9d6656-9a1a-445e-9b70-f3a9fad5038c-image-registry-private-configuration\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.204062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.204062    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df9d6656-9a1a-445e-9b70-f3a9fad5038c-trusted-ca\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.204410 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.204081    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df9d6656-9a1a-445e-9b70-f3a9fad5038c-ca-trust-extracted\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.204410 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.204096    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df9d6656-9a1a-445e-9b70-f3a9fad5038c-bound-sa-token\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.204410 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.204112    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z6ps\" (UniqueName: \"kubernetes.io/projected/df9d6656-9a1a-445e-9b70-f3a9fad5038c-kube-api-access-8z6ps\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.204410 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.204135    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df9d6656-9a1a-445e-9b70-f3a9fad5038c-registry-tls\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.204769 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.204748    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df9d6656-9a1a-445e-9b70-f3a9fad5038c-ca-trust-extracted\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.205096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.205070    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df9d6656-9a1a-445e-9b70-f3a9fad5038c-registry-certificates\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.205147 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.205133    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df9d6656-9a1a-445e-9b70-f3a9fad5038c-trusted-ca\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.206588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.206569    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df9d6656-9a1a-445e-9b70-f3a9fad5038c-image-registry-private-configuration\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.207162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.207139    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df9d6656-9a1a-445e-9b70-f3a9fad5038c-installation-pull-secrets\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.207236 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.207160    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df9d6656-9a1a-445e-9b70-f3a9fad5038c-registry-tls\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.232788 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.232766    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df9d6656-9a1a-445e-9b70-f3a9fad5038c-bound-sa-token\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.233294 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.233274    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z6ps\" (UniqueName: \"kubernetes.io/projected/df9d6656-9a1a-445e-9b70-f3a9fad5038c-kube-api-access-8z6ps\") pod \"image-registry-76c8cdc68c-8jdsl\" (UID: \"df9d6656-9a1a-445e-9b70-f3a9fad5038c\") " pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.253125 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.253101    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.566953 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.566925    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76c8cdc68c-8jdsl"]
Oct 30 00:05:03.570008 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:05:03.569981    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9d6656_9a1a_445e_9b70_f3a9fad5038c.slice/crio-2b9f7c1c76faead1ef4e485c8bf88ee5423bb0835c1eaf75068cb85641d32f85 WatchSource:0}: Error finding container 2b9f7c1c76faead1ef4e485c8bf88ee5423bb0835c1eaf75068cb85641d32f85: Status 404 returned error can't find the container with id 2b9f7c1c76faead1ef4e485c8bf88ee5423bb0835c1eaf75068cb85641d32f85
Oct 30 00:05:03.683778 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.683745    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl" event={"ID":"df9d6656-9a1a-445e-9b70-f3a9fad5038c","Type":"ContainerStarted","Data":"16e463d5893cf4878ec2a164dea5e846e3138f785de36ed43c98f368a4b6c5ce"}
Oct 30 00:05:03.683920 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.683791    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl" event={"ID":"df9d6656-9a1a-445e-9b70-f3a9fad5038c","Type":"ContainerStarted","Data":"2b9f7c1c76faead1ef4e485c8bf88ee5423bb0835c1eaf75068cb85641d32f85"}
Oct 30 00:05:03.683920 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.683897    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:03.727275 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:03.727222    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl" podStartSLOduration=1.727208219 podStartE2EDuration="1.727208219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:03.726495164 +0000 UTC m=+13.251789064" watchObservedRunningTime="2026-10-30 00:05:03.727208219 +0000 UTC m=+13.252502097"
Oct 30 00:05:03.898351 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:05:03.898275    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:05:05.659943 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:05.659906    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xzfnp"
Oct 30 00:05:09.510084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.510053    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7847f9c4d9-lpfsv"]
Oct 30 00:05:09.513115 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.513093    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.538899 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.538869    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 00:05:09.538899 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.538897    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 00:05:09.539082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.538917    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 00:05:09.539082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.538917    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 00:05:09.539082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.538976    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 00:05:09.539224 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.539210    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 00:05:09.539224 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.539221    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xz5vx\""
Oct 30 00:05:09.539422 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.539409    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 00:05:09.551292 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.551273    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7847f9c4d9-lpfsv"]
Oct 30 00:05:09.644272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.644246    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-oauth-serving-cert\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.644410 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.644284    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-console-config\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.644410 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.644309    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-serving-cert\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.644499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.644404    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edb34a91-bd43-430a-95a6-9784859c0b68-tmp\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.644499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.644435    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7sns\" (UniqueName: \"kubernetes.io/projected/edb34a91-bd43-430a-95a6-9784859c0b68-kube-api-access-r7sns\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.644499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.644458    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-oauth-config\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.644499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.644473    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-service-ca\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.744825 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.744800    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-oauth-serving-cert\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.744965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.744833    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-console-config\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.744965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.744859    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-serving-cert\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.744965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.744882    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edb34a91-bd43-430a-95a6-9784859c0b68-tmp\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.744965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.744900    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7sns\" (UniqueName: \"kubernetes.io/projected/edb34a91-bd43-430a-95a6-9784859c0b68-kube-api-access-r7sns\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.744965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.744919    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-oauth-config\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.744965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.744936    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-service-ca\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.745446 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.745412    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edb34a91-bd43-430a-95a6-9784859c0b68-tmp\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.745664 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.745637    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-oauth-serving-cert\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.745777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.745715    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-console-config\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.745777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.745744    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-service-ca\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.747925 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.747904    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-oauth-config\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.748008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.747946    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-serving-cert\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.767750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.767703    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7sns\" (UniqueName: \"kubernetes.io/projected/edb34a91-bd43-430a-95a6-9784859c0b68-kube-api-access-r7sns\") pod \"console-7847f9c4d9-lpfsv\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") " pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:09.823531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:09.823499    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:10.113356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:10.113269    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7847f9c4d9-lpfsv"]
Oct 30 00:05:10.117077 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:05:10.117050    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb34a91_bd43_430a_95a6_9784859c0b68.slice/crio-a74b843fbac4a458e5536220205f9d755c6b32183500bce5d680e887c9758f15 WatchSource:0}: Error finding container a74b843fbac4a458e5536220205f9d755c6b32183500bce5d680e887c9758f15: Status 404 returned error can't find the container with id a74b843fbac4a458e5536220205f9d755c6b32183500bce5d680e887c9758f15
Oct 30 00:05:10.705906 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:10.705865    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7847f9c4d9-lpfsv" event={"ID":"edb34a91-bd43-430a-95a6-9784859c0b68","Type":"ContainerStarted","Data":"a74b843fbac4a458e5536220205f9d755c6b32183500bce5d680e887c9758f15"}
Oct 30 00:05:10.978381 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:10.978349    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"]
Oct 30 00:05:10.980748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:10.980719    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:10.991691 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:10.991663    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-swwfx"]
Oct 30 00:05:10.994320 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:10.994285    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.002855 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.002832    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 00:05:11.010096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.010073    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 00:05:11.010204 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.010134    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 00:05:11.010204 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.010186    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 00:05:11.014398 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.014373    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 00:05:11.014398 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.014373    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 00:05:11.014559 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.014501    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gplj9\""
Oct 30 00:05:11.015316 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.015297    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"]
Oct 30 00:05:11.019489 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.019448    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 00:05:11.019489 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.019449    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 00:05:11.019607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.019507    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x8k7p\""
Oct 30 00:05:11.046480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.046462    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 00:05:11.153228 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153197    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44247589-aae2-437c-807f-4f36d318b7ca-root\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.153403 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153240    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5179b345-9698-4fb0-9628-fc487b250f56-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.153403 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153367    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-accelerators-collector-config\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.153523 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153414    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44247589-aae2-437c-807f-4f36d318b7ca-sys\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.153523 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153466    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44247589-aae2-437c-807f-4f36d318b7ca-metrics-client-ca\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.153682 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153526    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5179b345-9698-4fb0-9628-fc487b250f56-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.153682 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153556    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5179b345-9698-4fb0-9628-fc487b250f56-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.153682 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153588    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-wtmp\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.153682 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153616    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.153682 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153651    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fb2l\" (UniqueName: \"kubernetes.io/projected/44247589-aae2-437c-807f-4f36d318b7ca-kube-api-access-8fb2l\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.153924 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153682    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5179b345-9698-4fb0-9628-fc487b250f56-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.153924 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153708    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njwrh\" (UniqueName: \"kubernetes.io/projected/5179b345-9698-4fb0-9628-fc487b250f56-kube-api-access-njwrh\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.153924 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153732    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-textfile\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.153924 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153768    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-tls\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.153924 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.153830    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5179b345-9698-4fb0-9628-fc487b250f56-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.255167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255087    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5179b345-9698-4fb0-9628-fc487b250f56-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.255167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255145    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44247589-aae2-437c-807f-4f36d318b7ca-root\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.255377 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255175    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5179b345-9698-4fb0-9628-fc487b250f56-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.255377 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255202    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-accelerators-collector-config\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.255377 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255222    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44247589-aae2-437c-807f-4f36d318b7ca-root\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.255377 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255234    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44247589-aae2-437c-807f-4f36d318b7ca-sys\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.255377 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255289    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44247589-aae2-437c-807f-4f36d318b7ca-sys\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.255377 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255322    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44247589-aae2-437c-807f-4f36d318b7ca-metrics-client-ca\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.255377 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255353    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5179b345-9698-4fb0-9628-fc487b250f56-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.255377 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255380    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5179b345-9698-4fb0-9628-fc487b250f56-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.255762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255409    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-wtmp\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.255762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255436    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.255762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255468    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fb2l\" (UniqueName: \"kubernetes.io/projected/44247589-aae2-437c-807f-4f36d318b7ca-kube-api-access-8fb2l\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.255762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255500    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5179b345-9698-4fb0-9628-fc487b250f56-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.255762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255525    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njwrh\" (UniqueName: \"kubernetes.io/projected/5179b345-9698-4fb0-9628-fc487b250f56-kube-api-access-njwrh\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.255762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255548    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-textfile\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.256076 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255776    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-accelerators-collector-config\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.256076 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.255857    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-textfile\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.256200 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.256085    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44247589-aae2-437c-807f-4f36d318b7ca-metrics-client-ca\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.256251 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.256228    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-wtmp\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.256303 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.256257    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-tls\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.256494 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.256468    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5179b345-9698-4fb0-9628-fc487b250f56-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.256842 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.256821    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5179b345-9698-4fb0-9628-fc487b250f56-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.257383 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.257361    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5179b345-9698-4fb0-9628-fc487b250f56-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.258572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.258503    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5179b345-9698-4fb0-9628-fc487b250f56-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.258572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.258513    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5179b345-9698-4fb0-9628-fc487b250f56-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.259038 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.259002    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.260255 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.260233    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44247589-aae2-437c-807f-4f36d318b7ca-node-exporter-tls\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.279036 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.278995    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njwrh\" (UniqueName: \"kubernetes.io/projected/5179b345-9698-4fb0-9628-fc487b250f56-kube-api-access-njwrh\") pod \"kube-state-metrics-5dc8d7d89b-7k5ps\" (UID: \"5179b345-9698-4fb0-9628-fc487b250f56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.279149 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.279101    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fb2l\" (UniqueName: \"kubernetes.io/projected/44247589-aae2-437c-807f-4f36d318b7ca-kube-api-access-8fb2l\") pod \"node-exporter-swwfx\" (UID: \"44247589-aae2-437c-807f-4f36d318b7ca\") " pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.296930 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.296909    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"
Oct 30 00:05:11.309608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.309583    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-swwfx"
Oct 30 00:05:11.319674 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:05:11.319644    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44247589_aae2_437c_807f_4f36d318b7ca.slice/crio-a5cf7e1b0baab5223f51b0d629d74c9c31db78876138486544765e942897df44 WatchSource:0}: Error finding container a5cf7e1b0baab5223f51b0d629d74c9c31db78876138486544765e942897df44: Status 404 returned error can't find the container with id a5cf7e1b0baab5223f51b0d629d74c9c31db78876138486544765e942897df44
Oct 30 00:05:11.621468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.621319    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps"]
Oct 30 00:05:11.710369 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.710330    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swwfx" event={"ID":"44247589-aae2-437c-807f-4f36d318b7ca","Type":"ContainerStarted","Data":"a5cf7e1b0baab5223f51b0d629d74c9c31db78876138486544765e942897df44"}
Oct 30 00:05:11.711745 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:11.711717    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps" event={"ID":"5179b345-9698-4fb0-9628-fc487b250f56","Type":"ContainerStarted","Data":"cf7ef6c4f7d4b5cf1778e83ada4a5e190e18690c05a41245ad706cabc4b3e88a"}
Oct 30 00:05:12.718694 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:12.718605    4441 generic.go:356] "Generic (PLEG): container finished" podID="44247589-aae2-437c-807f-4f36d318b7ca" containerID="24048e4cb0fdea60acf107e857636b40989475af063dec5f85a978ce6bc61057" exitCode=0
Oct 30 00:05:12.718694 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:12.718674    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swwfx" event={"ID":"44247589-aae2-437c-807f-4f36d318b7ca","Type":"ContainerDied","Data":"24048e4cb0fdea60acf107e857636b40989475af063dec5f85a978ce6bc61057"}
Oct 30 00:05:13.724545 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:13.724504    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swwfx" event={"ID":"44247589-aae2-437c-807f-4f36d318b7ca","Type":"ContainerStarted","Data":"5c106c604309f0971e9fc6fdff98092ef423310d6d4dd7de2fdfdfb97ea58da4"}
Oct 30 00:05:13.725057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:13.724566    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swwfx" event={"ID":"44247589-aae2-437c-807f-4f36d318b7ca","Type":"ContainerStarted","Data":"e5f1b0b2a6d91f7fde1dfc78db8a4205cf88e688b2bb79e9411ef20c51fd2098"}
Oct 30 00:05:13.726635 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:13.726611    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps" event={"ID":"5179b345-9698-4fb0-9628-fc487b250f56","Type":"ContainerStarted","Data":"a5aee104ecf2cfee1b2d2375b25cfa55a5182a2b77e38f68e8fe72f9ababeb0f"}
Oct 30 00:05:13.726748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:13.726659    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps" event={"ID":"5179b345-9698-4fb0-9628-fc487b250f56","Type":"ContainerStarted","Data":"2d682905a0ce8cba84dfce23e005192b7d7333f1b3e18634abd1e9014a036c39"}
Oct 30 00:05:13.726748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:13.726677    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps" event={"ID":"5179b345-9698-4fb0-9628-fc487b250f56","Type":"ContainerStarted","Data":"cf0c4c27af7e4c9cd6b07cbed1f7111551183227835bcc8346b9a7b04acf6d9b"}
Oct 30 00:05:13.728083 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:13.728062    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7847f9c4d9-lpfsv" event={"ID":"edb34a91-bd43-430a-95a6-9784859c0b68","Type":"ContainerStarted","Data":"5c79fa4186df117bc35f248ebd96e4f3ced13a316b86458d3891d8259106c2f0"}
Oct 30 00:05:13.767923 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:13.767870    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-swwfx" podStartSLOduration=2.605910756 podStartE2EDuration="3.767850323s" totalImagesPullingTime="1.080269669s" totalInitContainerRuntime="81.669898ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:13.766721743 +0000 UTC m=+23.292015622" watchObservedRunningTime="2026-10-30 00:05:13.767850323 +0000 UTC m=+23.293144199"
Oct 30 00:05:13.807790 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:13.807737    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7847f9c4d9-lpfsv" podStartSLOduration=1.756434222 podStartE2EDuration="4.807724148s" totalImagesPullingTime="3.051289926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:13.806848745 +0000 UTC m=+23.332142622" watchObservedRunningTime="2026-10-30 00:05:13.807724148 +0000 UTC m=+23.333018025"
Oct 30 00:05:13.848100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:13.848052    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7k5ps" podStartSLOduration=2.438558112 podStartE2EDuration="3.848013102s" totalImagesPullingTime="1.40945499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:13.84784273 +0000 UTC m=+23.373136606" watchObservedRunningTime="2026-10-30 00:05:13.848013102 +0000 UTC m=+23.373306979"
Oct 30 00:05:17.950906 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.950869    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:05:17.954220 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.954202    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:17.968957 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.968932    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 00:05:17.968957 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.968963    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 00:05:17.968957 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.968970    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 00:05:17.969270 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.969082    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 00:05:17.969270 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.969131    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 00:05:17.969270 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.969156    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-v845r\""
Oct 30 00:05:17.969270 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.969202    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e6uu3k2pjufvn\""
Oct 30 00:05:17.969270 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.969204    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 00:05:17.980470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.980447    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 00:05:17.980633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.980535    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 00:05:17.980633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.980594    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 00:05:17.980633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.980619    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 00:05:17.980633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.980632    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 00:05:17.980633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.980651    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 00:05:17.980941 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:17.980790    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 00:05:18.004845 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.004809    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 00:05:18.004997 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.004974    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 00:05:18.005749 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.005730    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:05:18.105183 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105151    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105195    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105218    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105276    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105301    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105320    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-config\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105345    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-web-config\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105369    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105424    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105444    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105493    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105593    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk9t6\" (UniqueName: \"kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-kube-api-access-qk9t6\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105809 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105622    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-config-out\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105809 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105658    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105809 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105697    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105809 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105715    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105809 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105731    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105809 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105755    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.105809 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105786    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.106005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.105822    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.206886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.206832    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.206886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.206857    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.206886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.206878    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk9t6\" (UniqueName: \"kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-kube-api-access-qk9t6\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.207142 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.206895    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-config-out\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.207142 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.207016    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.207142 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.207064    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.207142 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.207090    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.207142 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.207117    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.207378 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.207144    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.207378 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.207169    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.207378 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.207195    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.207378 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.207224    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.207378 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.207273    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.209427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.208313    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.209427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.208308    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.209427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.208493    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.209427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.208536    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.209427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.208581    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-config\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.209427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.208651    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-web-config\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.209427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.208753    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.209427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.208921    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.209427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.209296    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.210826 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.210801    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.212128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.211204    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.212128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.211503    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.212128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.211605    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.212335 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.212264    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.213286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.213228    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.215466 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.214004    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.215466 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.214472    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.215466 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.214534    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-web-config\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.215466 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.214642    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-config-out\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.215466 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.215205    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.215741 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.215514    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.216801 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.216775    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.216801 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.216789    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.217088 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.217070    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.217387 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.217367    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-config\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.217794 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.217778    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.230391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.230365    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk9t6\" (UniqueName: \"kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-kube-api-access-qk9t6\") pod \"prometheus-k8s-0\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.270222 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.270199    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:18.606361 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:05:18.606329    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdb5e50_e931_4391_b203_f4886481e1ff.slice/crio-b97c64551d15573a853b444a75e4076a776e20963924d1ad513e06df0676eac7 WatchSource:0}: Error finding container b97c64551d15573a853b444a75e4076a776e20963924d1ad513e06df0676eac7: Status 404 returned error can't find the container with id b97c64551d15573a853b444a75e4076a776e20963924d1ad513e06df0676eac7
Oct 30 00:05:18.606645 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.606628    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:05:18.744904 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:18.744879    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerStarted","Data":"b97c64551d15573a853b444a75e4076a776e20963924d1ad513e06df0676eac7"}
Oct 30 00:05:19.749478 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:19.749447    4441 generic.go:356] "Generic (PLEG): container finished" podID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerID="a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6" exitCode=0
Oct 30 00:05:19.749759 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:19.749535    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerDied","Data":"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"}
Oct 30 00:05:19.824258 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:19.824191    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:19.824258 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:19.824255    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:19.828810 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:19.828789    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:20.758259 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:20.758230    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:23.766715 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:23.766684    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerStarted","Data":"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"}
Oct 30 00:05:23.767092 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:23.766745    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerStarted","Data":"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"}
Oct 30 00:05:24.692642 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:24.692609    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76c8cdc68c-8jdsl"
Oct 30 00:05:25.656256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:25.656227    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tr4w8"
Oct 30 00:05:26.780809 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:26.780781    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerStarted","Data":"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"}
Oct 30 00:05:26.781195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:26.780843    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerStarted","Data":"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"}
Oct 30 00:05:26.781195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:26.780871    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerStarted","Data":"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"}
Oct 30 00:05:26.781195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:26.780903    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerStarted","Data":"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"}
Oct 30 00:05:26.863614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:26.863567    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.041602558 podStartE2EDuration="9.863553837s" totalImagesPullingTime="6.805877582s" totalInitContainerRuntime="16.073697ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:26.86253308 +0000 UTC m=+36.387826956" watchObservedRunningTime="2026-10-30 00:05:26.863553837 +0000 UTC m=+36.388847713"
Oct 30 00:05:28.271335 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:28.271301    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:31.529480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:31.529438    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7847f9c4d9-lpfsv"]
Oct 30 00:05:33.897215 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:05:33.897176    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:05:41.940630 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:41.940593    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:41.958718 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:41.958687    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:42.851178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:42.851152    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:05:56.549091 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.549018    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7847f9c4d9-lpfsv" podUID="edb34a91-bd43-430a-95a6-9784859c0b68" containerName="console" containerID="cri-o://5c79fa4186df117bc35f248ebd96e4f3ced13a316b86458d3891d8259106c2f0" gracePeriod=15
Oct 30 00:05:56.789438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.789416    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:56.883654 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.883598    4441 generic.go:356] "Generic (PLEG): container finished" podID="edb34a91-bd43-430a-95a6-9784859c0b68" containerID="5c79fa4186df117bc35f248ebd96e4f3ced13a316b86458d3891d8259106c2f0" exitCode=0
Oct 30 00:05:56.883654 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.883638    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7847f9c4d9-lpfsv"
Oct 30 00:05:56.883654 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.883644    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7847f9c4d9-lpfsv" event={"ID":"edb34a91-bd43-430a-95a6-9784859c0b68","Type":"ContainerDied","Data":"5c79fa4186df117bc35f248ebd96e4f3ced13a316b86458d3891d8259106c2f0"}
Oct 30 00:05:56.883869 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.883682    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7847f9c4d9-lpfsv" event={"ID":"edb34a91-bd43-430a-95a6-9784859c0b68","Type":"ContainerDied","Data":"a74b843fbac4a458e5536220205f9d755c6b32183500bce5d680e887c9758f15"}
Oct 30 00:05:56.883869 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.883702    4441 scope.go:122] "RemoveContainer" containerID="5c79fa4186df117bc35f248ebd96e4f3ced13a316b86458d3891d8259106c2f0"
Oct 30 00:05:56.887930 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.887911    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-service-ca\") pod \"edb34a91-bd43-430a-95a6-9784859c0b68\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") "
Oct 30 00:05:56.888054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.887950    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb34a91-bd43-430a-95a6-9784859c0b68-tmp\" (UniqueName: \"kubernetes.io/empty-dir/edb34a91-bd43-430a-95a6-9784859c0b68-tmp\") pod \"edb34a91-bd43-430a-95a6-9784859c0b68\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") "
Oct 30 00:05:56.888054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.887986    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb34a91-bd43-430a-95a6-9784859c0b68-kube-api-access-r7sns\" (UniqueName: \"kubernetes.io/projected/edb34a91-bd43-430a-95a6-9784859c0b68-kube-api-access-r7sns\") pod \"edb34a91-bd43-430a-95a6-9784859c0b68\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") "
Oct 30 00:05:56.888054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.888037    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-serving-cert\") pod \"edb34a91-bd43-430a-95a6-9784859c0b68\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") "
Oct 30 00:05:56.888234 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.888090    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-oauth-serving-cert\") pod \"edb34a91-bd43-430a-95a6-9784859c0b68\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") "
Oct 30 00:05:56.888234 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.888134    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-console-config\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-console-config\") pod \"edb34a91-bd43-430a-95a6-9784859c0b68\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") "
Oct 30 00:05:56.888234 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.888173    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-oauth-config\") pod \"edb34a91-bd43-430a-95a6-9784859c0b68\" (UID: \"edb34a91-bd43-430a-95a6-9784859c0b68\") "
Oct 30 00:05:56.888446 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.888382    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb34a91-bd43-430a-95a6-9784859c0b68-tmp" pod "edb34a91-bd43-430a-95a6-9784859c0b68" (UID: "edb34a91-bd43-430a-95a6-9784859c0b68"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:05:56.888572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.888449    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-service-ca" pod "edb34a91-bd43-430a-95a6-9784859c0b68" (UID: "edb34a91-bd43-430a-95a6-9784859c0b68"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:05:56.888644 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.888568    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-oauth-serving-cert" pod "edb34a91-bd43-430a-95a6-9784859c0b68" (UID: "edb34a91-bd43-430a-95a6-9784859c0b68"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:05:56.888820 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.888785    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-console-config" pod "edb34a91-bd43-430a-95a6-9784859c0b68" (UID: "edb34a91-bd43-430a-95a6-9784859c0b68"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:05:56.890370 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.890321    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb34a91-bd43-430a-95a6-9784859c0b68-kube-api-access-r7sns" pod "edb34a91-bd43-430a-95a6-9784859c0b68" (UID: "edb34a91-bd43-430a-95a6-9784859c0b68"). InnerVolumeSpecName "kube-api-access-r7sns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:05:56.890472 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.890415    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-serving-cert" pod "edb34a91-bd43-430a-95a6-9784859c0b68" (UID: "edb34a91-bd43-430a-95a6-9784859c0b68"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:05:56.891129 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.891105    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-oauth-config" pod "edb34a91-bd43-430a-95a6-9784859c0b68" (UID: "edb34a91-bd43-430a-95a6-9784859c0b68"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:05:56.892260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.892243    4441 scope.go:122] "RemoveContainer" containerID="5c79fa4186df117bc35f248ebd96e4f3ced13a316b86458d3891d8259106c2f0"
Oct 30 00:05:56.892536 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:05:56.892517    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c79fa4186df117bc35f248ebd96e4f3ced13a316b86458d3891d8259106c2f0\": container with ID starting with 5c79fa4186df117bc35f248ebd96e4f3ced13a316b86458d3891d8259106c2f0 not found: ID does not exist" containerID="5c79fa4186df117bc35f248ebd96e4f3ced13a316b86458d3891d8259106c2f0"
Oct 30 00:05:56.892611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.892548    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c79fa4186df117bc35f248ebd96e4f3ced13a316b86458d3891d8259106c2f0"} err="failed to get container status \"5c79fa4186df117bc35f248ebd96e4f3ced13a316b86458d3891d8259106c2f0\": rpc error: code = NotFound desc = could not find container \"5c79fa4186df117bc35f248ebd96e4f3ced13a316b86458d3891d8259106c2f0\": container with ID starting with 5c79fa4186df117bc35f248ebd96e4f3ced13a316b86458d3891d8259106c2f0 not found: ID does not exist"
Oct 30 00:05:56.988669 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.988644    4441 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edb34a91-bd43-430a-95a6-9784859c0b68-tmp\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:05:56.988769 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.988668    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7sns\" (UniqueName: \"kubernetes.io/projected/edb34a91-bd43-430a-95a6-9784859c0b68-kube-api-access-r7sns\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:05:56.988769 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.988684    4441 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-serving-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:05:56.988769 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.988698    4441 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-oauth-serving-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:05:56.988769 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.988711    4441 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-console-config\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:05:56.988769 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.988723    4441 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edb34a91-bd43-430a-95a6-9784859c0b68-console-oauth-config\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:05:56.988769 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:56.988736    4441 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edb34a91-bd43-430a-95a6-9784859c0b68-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:05:57.225655 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:57.225631    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7847f9c4d9-lpfsv"]
Oct 30 00:05:57.237188 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:57.237160    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7847f9c4d9-lpfsv"]
Oct 30 00:05:58.605833 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:05:58.605797    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edb34a91-bd43-430a-95a6-9784859c0b68" path="/var/lib/kubelet/pods/edb34a91-bd43-430a-95a6-9784859c0b68/volumes"
Oct 30 00:06:03.896768 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:06:03.896729    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:06:06.556055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.556004    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:06:06.556493 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.556329    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="prometheus" containerID="cri-o://a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4" gracePeriod=600
Oct 30 00:06:06.556493 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.556362    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="kube-rbac-proxy" containerID="cri-o://2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1" gracePeriod=600
Oct 30 00:06:06.556493 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.556402    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="config-reloader" containerID="cri-o://0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308" gracePeriod=600
Oct 30 00:06:06.556493 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.556419    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="kube-rbac-proxy-web" containerID="cri-o://cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638" gracePeriod=600
Oct 30 00:06:06.556493 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.556398    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="thanos-sidecar" containerID="cri-o://18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79" gracePeriod=600
Oct 30 00:06:06.556748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.556402    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6" gracePeriod=600
Oct 30 00:06:06.795807 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.795782    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:06.861688 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.861612    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-web-config\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-web-config\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.861688 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.861644    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-serving-certs-ca-bundle\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.861688 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.861663    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-thanos-prometheus-http-client-file\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.861688 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.861685    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-tls-assets\" (UniqueName: \"kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-tls-assets\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.861716    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-db\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.861741    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-tls\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.861795    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.861840    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-kubelet-serving-ca-bundle\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.861873    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-2\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.861903    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.861935    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-grpc-tls\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.861964    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-config\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-config\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.861996    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-metrics-client-certs\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.862038    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-trusted-ca-bundle\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.862049    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-serving-certs-ca-bundle" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:06:06.862498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.862071    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-metrics-client-ca\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.862116    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-kube-rbac-proxy\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.862150    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-1\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.862194    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-kube-api-access-qk9t6\" (UniqueName: \"kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-kube-api-access-qk9t6\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.862223    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-config-out\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.862269    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-kubelet-serving-ca-bundle" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:06:06.862498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.862287    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-0\") pod \"fcdb5e50-e931-4391-b203-f4886481e1ff\" (UID: \"fcdb5e50-e931-4391-b203-f4886481e1ff\") "
Oct 30 00:06:06.862935 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.862508    4441 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.862935 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.862526    4441 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.862935 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.862731    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-2" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:06:06.863749 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.863426    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-metrics-client-ca" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:06:06.864264 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.864233    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-1" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:06:06.864517 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.864486    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-thanos-prometheus-http-client-file" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:06:06.864621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.864564    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-tls-assets" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:06:06.864621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.864605    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-trusted-ca-bundle" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:06:06.865406 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.865238    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-db" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:06:06.866245 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.866207    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-config" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:06:06.867579 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.867539    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-kube-api-access-qk9t6" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "kube-api-access-qk9t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:06:06.867772 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.867738    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-kube-rbac-proxy" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:06:06.868220 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.868176    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-metrics-client-certs" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:06:06.868786 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.868764    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-config-out" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:06:06.869417 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.869385    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-0" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:06:06.869819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.869792    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-grpc-tls" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:06:06.869912 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.869819    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-kube-rbac-proxy-web" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:06:06.869982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.869927    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-tls" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:06:06.871504 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.871467    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-thanos-sidecar-tls" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:06:06.879063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.879042    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-web-config" pod "fcdb5e50-e931-4391-b203-f4886481e1ff" (UID: "fcdb5e50-e931-4391-b203-f4886481e1ff"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:06:06.919305 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919281    4441 generic.go:356] "Generic (PLEG): container finished" podID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerID="a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6" exitCode=0
Oct 30 00:06:06.919305 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919303    4441 generic.go:356] "Generic (PLEG): container finished" podID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerID="2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1" exitCode=0
Oct 30 00:06:06.919430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919311    4441 generic.go:356] "Generic (PLEG): container finished" podID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerID="cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638" exitCode=0
Oct 30 00:06:06.919430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919317    4441 generic.go:356] "Generic (PLEG): container finished" podID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerID="18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79" exitCode=0
Oct 30 00:06:06.919430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919322    4441 generic.go:356] "Generic (PLEG): container finished" podID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerID="0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308" exitCode=0
Oct 30 00:06:06.919430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919327    4441 generic.go:356] "Generic (PLEG): container finished" podID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerID="a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4" exitCode=0
Oct 30 00:06:06.919430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919345    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerDied","Data":"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"}
Oct 30 00:06:06.919430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919359    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:06.919430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919400    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerDied","Data":"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"}
Oct 30 00:06:06.919430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919426    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerDied","Data":"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"}
Oct 30 00:06:06.919711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919453    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerDied","Data":"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"}
Oct 30 00:06:06.919711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919471    4441 scope.go:122] "RemoveContainer" containerID="a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"
Oct 30 00:06:06.919711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919476    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerDied","Data":"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"}
Oct 30 00:06:06.919711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919578    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerDied","Data":"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"}
Oct 30 00:06:06.919711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.919603    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcdb5e50-e931-4391-b203-f4886481e1ff","Type":"ContainerDied","Data":"b97c64551d15573a853b444a75e4076a776e20963924d1ad513e06df0676eac7"}
Oct 30 00:06:06.927422 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.927400    4441 scope.go:122] "RemoveContainer" containerID="2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"
Oct 30 00:06:06.934255 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.934239    4441 scope.go:122] "RemoveContainer" containerID="cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"
Oct 30 00:06:06.941072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.941056    4441 scope.go:122] "RemoveContainer" containerID="18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"
Oct 30 00:06:06.946813 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.946798    4441 scope.go:122] "RemoveContainer" containerID="0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"
Oct 30 00:06:06.952782 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.952766    4441 scope.go:122] "RemoveContainer" containerID="a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"
Oct 30 00:06:06.959315 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.959301    4441 scope.go:122] "RemoveContainer" containerID="a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"
Oct 30 00:06:06.963397 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963377    4441 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-kube-rbac-proxy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963397 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963398    4441 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963408    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk9t6\" (UniqueName: \"kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-kube-api-access-qk9t6\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963416    4441 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-config-out\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963426    4441 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963435    4441 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-web-config\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963444    4441 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963453    4441 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcdb5e50-e931-4391-b203-f4886481e1ff-tls-assets\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963461    4441 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-db\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963469    4441 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963478    4441 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963487    4441 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963496    4441 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963504    4441 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-grpc-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963512    4441 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-config\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963520    4441 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcdb5e50-e931-4391-b203-f4886481e1ff-secret-metrics-client-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963528    4441 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.963882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.963536    4441 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcdb5e50-e931-4391-b203-f4886481e1ff-configmap-metrics-client-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:06:06.965439 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.965427    4441 scope.go:122] "RemoveContainer" containerID="a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"
Oct 30 00:06:06.965677 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:06:06.965660    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6\": container with ID starting with a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6 not found: ID does not exist" containerID="a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"
Oct 30 00:06:06.965713 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.965686    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"} err="failed to get container status \"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6\": rpc error: code = NotFound desc = could not find container \"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6\": container with ID starting with a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6 not found: ID does not exist"
Oct 30 00:06:06.965713 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.965701    4441 scope.go:122] "RemoveContainer" containerID="2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"
Oct 30 00:06:06.965923 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:06:06.965909    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1\": container with ID starting with 2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1 not found: ID does not exist" containerID="2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"
Oct 30 00:06:06.965967 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.965928    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"} err="failed to get container status \"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1\": rpc error: code = NotFound desc = could not find container \"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1\": container with ID starting with 2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1 not found: ID does not exist"
Oct 30 00:06:06.965967 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.965940    4441 scope.go:122] "RemoveContainer" containerID="cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"
Oct 30 00:06:06.966168 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:06:06.966147    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638\": container with ID starting with cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638 not found: ID does not exist" containerID="cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"
Oct 30 00:06:06.966233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.966175    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"} err="failed to get container status \"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638\": rpc error: code = NotFound desc = could not find container \"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638\": container with ID starting with cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638 not found: ID does not exist"
Oct 30 00:06:06.966233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.966194    4441 scope.go:122] "RemoveContainer" containerID="18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"
Oct 30 00:06:06.966401 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:06:06.966386    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79\": container with ID starting with 18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79 not found: ID does not exist" containerID="18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"
Oct 30 00:06:06.966442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.966405    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"} err="failed to get container status \"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79\": rpc error: code = NotFound desc = could not find container \"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79\": container with ID starting with 18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79 not found: ID does not exist"
Oct 30 00:06:06.966442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.966420    4441 scope.go:122] "RemoveContainer" containerID="0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"
Oct 30 00:06:06.966599 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:06:06.966585    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308\": container with ID starting with 0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308 not found: ID does not exist" containerID="0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"
Oct 30 00:06:06.966634 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.966602    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"} err="failed to get container status \"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308\": rpc error: code = NotFound desc = could not find container \"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308\": container with ID starting with 0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308 not found: ID does not exist"
Oct 30 00:06:06.966634 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.966614    4441 scope.go:122] "RemoveContainer" containerID="a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"
Oct 30 00:06:06.966824 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:06:06.966808    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4\": container with ID starting with a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4 not found: ID does not exist" containerID="a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"
Oct 30 00:06:06.966868 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.966828    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"} err="failed to get container status \"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4\": rpc error: code = NotFound desc = could not find container \"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4\": container with ID starting with a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4 not found: ID does not exist"
Oct 30 00:06:06.966868 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.966839    4441 scope.go:122] "RemoveContainer" containerID="a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"
Oct 30 00:06:06.967053 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:06:06.967036    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6\": container with ID starting with a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6 not found: ID does not exist" containerID="a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"
Oct 30 00:06:06.967101 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.967055    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"} err="failed to get container status \"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6\": rpc error: code = NotFound desc = could not find container \"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6\": container with ID starting with a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6 not found: ID does not exist"
Oct 30 00:06:06.967101 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.967066    4441 scope.go:122] "RemoveContainer" containerID="a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"
Oct 30 00:06:06.967311 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.967296    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"} err="failed to get container status \"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6\": rpc error: code = NotFound desc = could not find container \"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6\": container with ID starting with a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6 not found: ID does not exist"
Oct 30 00:06:06.967311 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.967310    4441 scope.go:122] "RemoveContainer" containerID="2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"
Oct 30 00:06:06.967490 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.967474    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"} err="failed to get container status \"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1\": rpc error: code = NotFound desc = could not find container \"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1\": container with ID starting with 2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1 not found: ID does not exist"
Oct 30 00:06:06.967531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.967490    4441 scope.go:122] "RemoveContainer" containerID="cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"
Oct 30 00:06:06.967708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.967686    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"} err="failed to get container status \"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638\": rpc error: code = NotFound desc = could not find container \"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638\": container with ID starting with cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638 not found: ID does not exist"
Oct 30 00:06:06.967764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.967709    4441 scope.go:122] "RemoveContainer" containerID="18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"
Oct 30 00:06:06.967925 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.967909    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"} err="failed to get container status \"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79\": rpc error: code = NotFound desc = could not find container \"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79\": container with ID starting with 18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79 not found: ID does not exist"
Oct 30 00:06:06.967984 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.967924    4441 scope.go:122] "RemoveContainer" containerID="0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"
Oct 30 00:06:06.968216 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.968199    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"} err="failed to get container status \"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308\": rpc error: code = NotFound desc = could not find container \"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308\": container with ID starting with 0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308 not found: ID does not exist"
Oct 30 00:06:06.968216 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.968215    4441 scope.go:122] "RemoveContainer" containerID="a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"
Oct 30 00:06:06.968410 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.968389    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"} err="failed to get container status \"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4\": rpc error: code = NotFound desc = could not find container \"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4\": container with ID starting with a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4 not found: ID does not exist"
Oct 30 00:06:06.968459 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.968411    4441 scope.go:122] "RemoveContainer" containerID="a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"
Oct 30 00:06:06.968618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.968602    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"} err="failed to get container status \"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6\": rpc error: code = NotFound desc = could not find container \"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6\": container with ID starting with a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6 not found: ID does not exist"
Oct 30 00:06:06.968661 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.968618    4441 scope.go:122] "RemoveContainer" containerID="a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"
Oct 30 00:06:06.968805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.968790    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"} err="failed to get container status \"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6\": rpc error: code = NotFound desc = could not find container \"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6\": container with ID starting with a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6 not found: ID does not exist"
Oct 30 00:06:06.968842 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.968805    4441 scope.go:122] "RemoveContainer" containerID="2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"
Oct 30 00:06:06.968980 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.968965    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"} err="failed to get container status \"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1\": rpc error: code = NotFound desc = could not find container \"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1\": container with ID starting with 2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1 not found: ID does not exist"
Oct 30 00:06:06.968980 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.968979    4441 scope.go:122] "RemoveContainer" containerID="cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"
Oct 30 00:06:06.969211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.969194    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"} err="failed to get container status \"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638\": rpc error: code = NotFound desc = could not find container \"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638\": container with ID starting with cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638 not found: ID does not exist"
Oct 30 00:06:06.969258 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.969211    4441 scope.go:122] "RemoveContainer" containerID="18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"
Oct 30 00:06:06.969419 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.969398    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"} err="failed to get container status \"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79\": rpc error: code = NotFound desc = could not find container \"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79\": container with ID starting with 18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79 not found: ID does not exist"
Oct 30 00:06:06.969456 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.969419    4441 scope.go:122] "RemoveContainer" containerID="0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"
Oct 30 00:06:06.969627 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.969611    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"} err="failed to get container status \"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308\": rpc error: code = NotFound desc = could not find container \"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308\": container with ID starting with 0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308 not found: ID does not exist"
Oct 30 00:06:06.969666 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.969627    4441 scope.go:122] "RemoveContainer" containerID="a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"
Oct 30 00:06:06.969838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.969823    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"} err="failed to get container status \"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4\": rpc error: code = NotFound desc = could not find container \"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4\": container with ID starting with a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4 not found: ID does not exist"
Oct 30 00:06:06.969838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.969837    4441 scope.go:122] "RemoveContainer" containerID="a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"
Oct 30 00:06:06.970018 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.970003    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"} err="failed to get container status \"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6\": rpc error: code = NotFound desc = could not find container \"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6\": container with ID starting with a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6 not found: ID does not exist"
Oct 30 00:06:06.970018 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.970016    4441 scope.go:122] "RemoveContainer" containerID="a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"
Oct 30 00:06:06.970265 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.970246    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"} err="failed to get container status \"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6\": rpc error: code = NotFound desc = could not find container \"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6\": container with ID starting with a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6 not found: ID does not exist"
Oct 30 00:06:06.970309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.970266    4441 scope.go:122] "RemoveContainer" containerID="2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"
Oct 30 00:06:06.970476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.970461    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"} err="failed to get container status \"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1\": rpc error: code = NotFound desc = could not find container \"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1\": container with ID starting with 2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1 not found: ID does not exist"
Oct 30 00:06:06.970515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.970476    4441 scope.go:122] "RemoveContainer" containerID="cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"
Oct 30 00:06:06.970673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.970657    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"} err="failed to get container status \"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638\": rpc error: code = NotFound desc = could not find container \"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638\": container with ID starting with cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638 not found: ID does not exist"
Oct 30 00:06:06.970673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.970672    4441 scope.go:122] "RemoveContainer" containerID="18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"
Oct 30 00:06:06.970866 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.970851    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"} err="failed to get container status \"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79\": rpc error: code = NotFound desc = could not find container \"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79\": container with ID starting with 18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79 not found: ID does not exist"
Oct 30 00:06:06.970866 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.970866    4441 scope.go:122] "RemoveContainer" containerID="0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"
Oct 30 00:06:06.971108 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.971092    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"} err="failed to get container status \"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308\": rpc error: code = NotFound desc = could not find container \"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308\": container with ID starting with 0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308 not found: ID does not exist"
Oct 30 00:06:06.971171 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.971108    4441 scope.go:122] "RemoveContainer" containerID="a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"
Oct 30 00:06:06.971296 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.971279    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"} err="failed to get container status \"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4\": rpc error: code = NotFound desc = could not find container \"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4\": container with ID starting with a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4 not found: ID does not exist"
Oct 30 00:06:06.971334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.971297    4441 scope.go:122] "RemoveContainer" containerID="a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"
Oct 30 00:06:06.971509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.971486    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"} err="failed to get container status \"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6\": rpc error: code = NotFound desc = could not find container \"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6\": container with ID starting with a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6 not found: ID does not exist"
Oct 30 00:06:06.971553 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.971510    4441 scope.go:122] "RemoveContainer" containerID="a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"
Oct 30 00:06:06.971767 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.971750    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"} err="failed to get container status \"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6\": rpc error: code = NotFound desc = could not find container \"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6\": container with ID starting with a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6 not found: ID does not exist"
Oct 30 00:06:06.971808 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.971768    4441 scope.go:122] "RemoveContainer" containerID="2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"
Oct 30 00:06:06.972003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.971986    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"} err="failed to get container status \"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1\": rpc error: code = NotFound desc = could not find container \"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1\": container with ID starting with 2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1 not found: ID does not exist"
Oct 30 00:06:06.972084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.972004    4441 scope.go:122] "RemoveContainer" containerID="cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"
Oct 30 00:06:06.972256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.972239    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"} err="failed to get container status \"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638\": rpc error: code = NotFound desc = could not find container \"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638\": container with ID starting with cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638 not found: ID does not exist"
Oct 30 00:06:06.972297 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.972256    4441 scope.go:122] "RemoveContainer" containerID="18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"
Oct 30 00:06:06.972484 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.972467    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"} err="failed to get container status \"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79\": rpc error: code = NotFound desc = could not find container \"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79\": container with ID starting with 18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79 not found: ID does not exist"
Oct 30 00:06:06.972536 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.972484    4441 scope.go:122] "RemoveContainer" containerID="0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"
Oct 30 00:06:06.972679 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.972662    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"} err="failed to get container status \"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308\": rpc error: code = NotFound desc = could not find container \"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308\": container with ID starting with 0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308 not found: ID does not exist"
Oct 30 00:06:06.972725 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.972680    4441 scope.go:122] "RemoveContainer" containerID="a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"
Oct 30 00:06:06.972894 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.972877    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"} err="failed to get container status \"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4\": rpc error: code = NotFound desc = could not find container \"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4\": container with ID starting with a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4 not found: ID does not exist"
Oct 30 00:06:06.972939 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.972894    4441 scope.go:122] "RemoveContainer" containerID="a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"
Oct 30 00:06:06.973104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.973078    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"} err="failed to get container status \"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6\": rpc error: code = NotFound desc = could not find container \"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6\": container with ID starting with a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6 not found: ID does not exist"
Oct 30 00:06:06.973104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.973103    4441 scope.go:122] "RemoveContainer" containerID="a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"
Oct 30 00:06:06.973271 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.973251    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6"} err="failed to get container status \"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6\": rpc error: code = NotFound desc = could not find container \"a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6\": container with ID starting with a76d795348596ea49aac608872b479e117a3c8284874bb1af681b78813d2c4d6 not found: ID does not exist"
Oct 30 00:06:06.973314 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.973271    4441 scope.go:122] "RemoveContainer" containerID="2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"
Oct 30 00:06:06.973491 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.973471    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1"} err="failed to get container status \"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1\": rpc error: code = NotFound desc = could not find container \"2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1\": container with ID starting with 2f84db2f61331cae208d573d5d6458f8f0b762b84859f3d7567fb3f51e3da7f1 not found: ID does not exist"
Oct 30 00:06:06.973537 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.973491    4441 scope.go:122] "RemoveContainer" containerID="cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"
Oct 30 00:06:06.973690 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.973669    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638"} err="failed to get container status \"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638\": rpc error: code = NotFound desc = could not find container \"cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638\": container with ID starting with cdd1bc8c2dd1a129e8395e5961dd874fd5bf4b6b4123496b8850b5f949197638 not found: ID does not exist"
Oct 30 00:06:06.973731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.973691    4441 scope.go:122] "RemoveContainer" containerID="18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"
Oct 30 00:06:06.973907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.973891    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79"} err="failed to get container status \"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79\": rpc error: code = NotFound desc = could not find container \"18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79\": container with ID starting with 18233fc9c122cd5ace3353a3dc8b13875ea3630bb4730377fa9727f03fe42a79 not found: ID does not exist"
Oct 30 00:06:06.973949 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.973908    4441 scope.go:122] "RemoveContainer" containerID="0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"
Oct 30 00:06:06.974128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.974112    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308"} err="failed to get container status \"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308\": rpc error: code = NotFound desc = could not find container \"0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308\": container with ID starting with 0778e5db83861231ca730399f3dd057a6f3d8e0926057f19a2ddaa738cc6b308 not found: ID does not exist"
Oct 30 00:06:06.974172 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.974129    4441 scope.go:122] "RemoveContainer" containerID="a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"
Oct 30 00:06:06.974311 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.974293    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4"} err="failed to get container status \"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4\": rpc error: code = NotFound desc = could not find container \"a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4\": container with ID starting with a25148eff5578b45bae36f142c1ce32d8a93c669c5673e660da22b899d41c4b4 not found: ID does not exist"
Oct 30 00:06:06.974356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.974311    4441 scope.go:122] "RemoveContainer" containerID="a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"
Oct 30 00:06:06.974537 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.974517    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6"} err="failed to get container status \"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6\": rpc error: code = NotFound desc = could not find container \"a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6\": container with ID starting with a771b2aaf15eb52336e32011c22569a1cd98ede2a266de2554fcfbf077093ac6 not found: ID does not exist"
Oct 30 00:06:06.977008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.976989    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:06:06.981842 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:06.981822    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:06:07.012996 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.012959    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:06:07.013278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013262    4441 cpu_manager.go:414] "containerMap: removing container" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="prometheus"
Oct 30 00:06:07.013354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013281    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="prometheus"
Oct 30 00:06:07.013354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013296    4441 cpu_manager.go:414] "containerMap: removing container" podUID="edb34a91-bd43-430a-95a6-9784859c0b68" containerName="console"
Oct 30 00:06:07.013354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013303    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb34a91-bd43-430a-95a6-9784859c0b68" containerName="console"
Oct 30 00:06:07.013354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013318    4441 cpu_manager.go:414] "containerMap: removing container" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="init-config-reloader"
Oct 30 00:06:07.013354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013326    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="init-config-reloader"
Oct 30 00:06:07.013354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013336    4441 cpu_manager.go:414] "containerMap: removing container" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="config-reloader"
Oct 30 00:06:07.013354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013343    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="config-reloader"
Oct 30 00:06:07.013354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013352    4441 cpu_manager.go:414] "containerMap: removing container" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="kube-rbac-proxy"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013359    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="kube-rbac-proxy"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013369    4441 cpu_manager.go:414] "containerMap: removing container" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="kube-rbac-proxy-thanos"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013376    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="kube-rbac-proxy-thanos"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013390    4441 cpu_manager.go:414] "containerMap: removing container" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="kube-rbac-proxy-web"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013397    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="kube-rbac-proxy-web"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013415    4441 cpu_manager.go:414] "containerMap: removing container" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="thanos-sidecar"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013423    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="thanos-sidecar"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013488    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="thanos-sidecar"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013502    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="prometheus"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013513    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb34a91-bd43-430a-95a6-9784859c0b68" containerName="console"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013523    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="config-reloader"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013531    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="kube-rbac-proxy"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013540    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="kube-rbac-proxy-thanos"
Oct 30 00:06:07.013734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.013548    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" containerName="kube-rbac-proxy-web"
Oct 30 00:06:07.017248 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.017230    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.032731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.032708    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 00:06:07.032819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.032750    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 00:06:07.032819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.032777    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 00:06:07.033327 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.033310    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 00:06:07.033409 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.033394    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 00:06:07.033462 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.033442    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 00:06:07.033525 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.033461    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 00:06:07.033525 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.033454    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 00:06:07.033525 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.033495    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 00:06:07.033525 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.033514    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 00:06:07.033665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.033554    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 00:06:07.033665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.033565    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 00:06:07.033665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.033652    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-v845r\""
Oct 30 00:06:07.033758 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.033701    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e6uu3k2pjufvn\""
Oct 30 00:06:07.033758 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.033722    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 00:06:07.045445 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.045429    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 00:06:07.045587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.045559    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 00:06:07.047063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.047040    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:06:07.064365 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064340    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064371    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064390    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064412    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064458    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064670 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064534    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-config\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064670 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064578    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4v89\" (UniqueName: \"kubernetes.io/projected/8e9581a9-4681-4cfa-ae1a-bc832915c956-kube-api-access-g4v89\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064670 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064608    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064670 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064642    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064674    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e9581a9-4681-4cfa-ae1a-bc832915c956-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064748    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064781    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e9581a9-4681-4cfa-ae1a-bc832915c956-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064797    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064976 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064862    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064976 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064894    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.064976 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064956    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.065094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.064978    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.065094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.065003    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.065094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.065059    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.065199 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.065113    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.165944    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.165983    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166011    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166250 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166060    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166250 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166096    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166250 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166232    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166272    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-config\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166301    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4v89\" (UniqueName: \"kubernetes.io/projected/8e9581a9-4681-4cfa-ae1a-bc832915c956-kube-api-access-g4v89\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166328    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166361    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166387    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e9581a9-4681-4cfa-ae1a-bc832915c956-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166413    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166440    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e9581a9-4681-4cfa-ae1a-bc832915c956-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166466    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166503    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166533    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166569    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166594    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.166631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166625    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.167097 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166661    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.167097 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166729    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.167097 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.166932    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.167862 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.167272    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.167862 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.167441    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.167862 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.167540    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.167862 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.167714    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.167862 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.167834    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.169923 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.169899    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e9581a9-4681-4cfa-ae1a-bc832915c956-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.170044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.170006    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.171748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.170223    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.171748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.170364    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e9581a9-4681-4cfa-ae1a-bc832915c956-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.171748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.170584    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.171748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.170621    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-config\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.171748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.171133    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.172449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.172417    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.172917 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.172893    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.173827 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.173794    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.175613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.174007    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e9581a9-4681-4cfa-ae1a-bc832915c956-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.175613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.174404    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e9581a9-4681-4cfa-ae1a-bc832915c956-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.193192 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.193172    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4v89\" (UniqueName: \"kubernetes.io/projected/8e9581a9-4681-4cfa-ae1a-bc832915c956-kube-api-access-g4v89\") pod \"prometheus-k8s-0\" (UID: \"8e9581a9-4681-4cfa-ae1a-bc832915c956\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.333817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.333793    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:07.649889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.649863    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:06:07.653193 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:06:07.653171    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9581a9_4681_4cfa_ae1a_bc832915c956.slice/crio-e46b1d3c1c33bbd8eb2ed216f6ffcddedbe71b280fc729f8a41765a6e4e13baf WatchSource:0}: Error finding container e46b1d3c1c33bbd8eb2ed216f6ffcddedbe71b280fc729f8a41765a6e4e13baf: Status 404 returned error can't find the container with id e46b1d3c1c33bbd8eb2ed216f6ffcddedbe71b280fc729f8a41765a6e4e13baf
Oct 30 00:06:07.923866 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.923796    4441 generic.go:356] "Generic (PLEG): container finished" podID="8e9581a9-4681-4cfa-ae1a-bc832915c956" containerID="32e8022de94917ff4d6e6c7df85f74fd657d4a06f903051ce84f776d6461b059" exitCode=0
Oct 30 00:06:07.923980 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.923886    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e9581a9-4681-4cfa-ae1a-bc832915c956","Type":"ContainerDied","Data":"32e8022de94917ff4d6e6c7df85f74fd657d4a06f903051ce84f776d6461b059"}
Oct 30 00:06:07.923980 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:07.923963    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e9581a9-4681-4cfa-ae1a-bc832915c956","Type":"ContainerStarted","Data":"e46b1d3c1c33bbd8eb2ed216f6ffcddedbe71b280fc729f8a41765a6e4e13baf"}
Oct 30 00:06:08.606669 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:08.606643    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcdb5e50-e931-4391-b203-f4886481e1ff" path="/var/lib/kubelet/pods/fcdb5e50-e931-4391-b203-f4886481e1ff/volumes"
Oct 30 00:06:08.931916 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:08.931856    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e9581a9-4681-4cfa-ae1a-bc832915c956","Type":"ContainerStarted","Data":"e6571eed966fd12d7c99632305b84dd0c58f8860fe7bc8f903f984930767c3bf"}
Oct 30 00:06:08.932340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:08.931931    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e9581a9-4681-4cfa-ae1a-bc832915c956","Type":"ContainerStarted","Data":"d510dc6d1a90ea141965df12acf991b55cbb9c15085b52e46f1262c9c181fea5"}
Oct 30 00:06:08.932340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:08.931974    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e9581a9-4681-4cfa-ae1a-bc832915c956","Type":"ContainerStarted","Data":"a7d98e76fe8793560d3f6667c79400ef0a0e683fe443851693853c7b77ea3a66"}
Oct 30 00:06:08.932340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:08.932016    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e9581a9-4681-4cfa-ae1a-bc832915c956","Type":"ContainerStarted","Data":"7a4dad01a15d302efa0e6903ff441f2589e5b7d2a55d2c3a8222e19379879df4"}
Oct 30 00:06:08.932340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:08.932068    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e9581a9-4681-4cfa-ae1a-bc832915c956","Type":"ContainerStarted","Data":"de775545afd267ac0190fd543fb1fe169bd646499a33b91fb56dc7d9d586d709"}
Oct 30 00:06:08.932340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:08.932107    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e9581a9-4681-4cfa-ae1a-bc832915c956","Type":"ContainerStarted","Data":"017c201e232e5d41e5dae07e4d018da51a26d02fd4ea3b6476d21f60bac06140"}
Oct 30 00:06:08.999315 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:08.999263    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.984858905 podStartE2EDuration="2.999245479s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.386574ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:06:08.998091452 +0000 UTC m=+78.523385326" watchObservedRunningTime="2026-10-30 00:06:08.999245479 +0000 UTC m=+78.524539357"
Oct 30 00:06:12.334204 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:06:12.334165    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:06:33.896623 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:06:33.896587    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:07:03.897082 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:07:03.897044    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:07:07.334222 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:07:07.334194    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:07:07.349090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:07:07.349066    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:07:08.134494 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:07:08.134467    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:07:33.896379 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:07:33.896347    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:08:03.897323 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:08:03.897283    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:08:33.898820 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:08:33.898773    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:08:35.819810 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:35.819784    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"]
Oct 30 00:08:35.822379 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:35.822361    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"
Oct 30 00:08:35.847964 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:35.847932    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-4tx68\""
Oct 30 00:08:35.848770 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:35.848752    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"]
Oct 30 00:08:35.865545 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:35.865522    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 00:08:35.865638 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:35.865546    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 00:08:35.986095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:35.986062    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh\" (UID: \"654a9dfa-aba3-458e-87f4-bd73305627fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"
Oct 30 00:08:35.986253 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:35.986105    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6ct\" (UniqueName: \"kubernetes.io/projected/654a9dfa-aba3-458e-87f4-bd73305627fe-kube-api-access-pg6ct\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh\" (UID: \"654a9dfa-aba3-458e-87f4-bd73305627fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"
Oct 30 00:08:35.986253 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:35.986162    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh\" (UID: \"654a9dfa-aba3-458e-87f4-bd73305627fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"
Oct 30 00:08:36.087114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:36.087046    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh\" (UID: \"654a9dfa-aba3-458e-87f4-bd73305627fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"
Oct 30 00:08:36.087237 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:36.087128    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh\" (UID: \"654a9dfa-aba3-458e-87f4-bd73305627fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"
Oct 30 00:08:36.087237 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:36.087162    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6ct\" (UniqueName: \"kubernetes.io/projected/654a9dfa-aba3-458e-87f4-bd73305627fe-kube-api-access-pg6ct\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh\" (UID: \"654a9dfa-aba3-458e-87f4-bd73305627fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"
Oct 30 00:08:36.087397 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:36.087379    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh\" (UID: \"654a9dfa-aba3-458e-87f4-bd73305627fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"
Oct 30 00:08:36.087464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:36.087435    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh\" (UID: \"654a9dfa-aba3-458e-87f4-bd73305627fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"
Oct 30 00:08:36.109513 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:36.109493    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6ct\" (UniqueName: \"kubernetes.io/projected/654a9dfa-aba3-458e-87f4-bd73305627fe-kube-api-access-pg6ct\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh\" (UID: \"654a9dfa-aba3-458e-87f4-bd73305627fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"
Oct 30 00:08:36.134291 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:36.134265    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"
Oct 30 00:08:36.425278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:36.425176    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"]
Oct 30 00:08:36.427944 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:08:36.427919    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654a9dfa_aba3_458e_87f4_bd73305627fe.slice/crio-b49545db95675c57ef13b67a3636c8d9d0448a707836d400ef5269efc18734d1 WatchSource:0}: Error finding container b49545db95675c57ef13b67a3636c8d9d0448a707836d400ef5269efc18734d1: Status 404 returned error can't find the container with id b49545db95675c57ef13b67a3636c8d9d0448a707836d400ef5269efc18734d1
Oct 30 00:08:37.398314 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:37.398234    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh" event={"ID":"654a9dfa-aba3-458e-87f4-bd73305627fe","Type":"ContainerStarted","Data":"b49545db95675c57ef13b67a3636c8d9d0448a707836d400ef5269efc18734d1"}
Oct 30 00:08:42.417322 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:42.417284    4441 generic.go:356] "Generic (PLEG): container finished" podID="654a9dfa-aba3-458e-87f4-bd73305627fe" containerID="d11aab1478d6baa6b435578806a36df9cf543818226634fad0a609e7be4a574d" exitCode=0
Oct 30 00:08:42.417722 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:42.417339    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh" event={"ID":"654a9dfa-aba3-458e-87f4-bd73305627fe","Type":"ContainerDied","Data":"d11aab1478d6baa6b435578806a36df9cf543818226634fad0a609e7be4a574d"}
Oct 30 00:08:43.422746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:43.422710    4441 generic.go:356] "Generic (PLEG): container finished" podID="654a9dfa-aba3-458e-87f4-bd73305627fe" containerID="74b09928ec8809cf14e54dac774fa6b9079f6467cadd7bbe5518042b078985e1" exitCode=0
Oct 30 00:08:43.423138 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:43.422753    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh" event={"ID":"654a9dfa-aba3-458e-87f4-bd73305627fe","Type":"ContainerDied","Data":"74b09928ec8809cf14e54dac774fa6b9079f6467cadd7bbe5518042b078985e1"}
Oct 30 00:08:49.446707 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:49.446676    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh" event={"ID":"654a9dfa-aba3-458e-87f4-bd73305627fe","Type":"ContainerStarted","Data":"369d45e2a1ff4f9251e958d31ad0c830f0bccf27973d81290984031d6b1b8596"}
Oct 30 00:08:49.487553 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:49.487502    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh" podStartSLOduration=2.956750793 podStartE2EDuration="14.487487626s" totalImagesPullingTime="11.530736833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:08:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:08:49.487141255 +0000 UTC m=+239.012435132" watchObservedRunningTime="2026-10-30 00:08:49.487487626 +0000 UTC m=+239.012781498"
Oct 30 00:08:50.452529 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:50.452493    4441 generic.go:356] "Generic (PLEG): container finished" podID="654a9dfa-aba3-458e-87f4-bd73305627fe" containerID="369d45e2a1ff4f9251e958d31ad0c830f0bccf27973d81290984031d6b1b8596" exitCode=0
Oct 30 00:08:50.452891 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:50.452563    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh" event={"ID":"654a9dfa-aba3-458e-87f4-bd73305627fe","Type":"ContainerDied","Data":"369d45e2a1ff4f9251e958d31ad0c830f0bccf27973d81290984031d6b1b8596"}
Oct 30 00:08:51.572364 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:51.572338    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"
Oct 30 00:08:51.598580 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:51.598553    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-bundle\" (UniqueName: \"kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-bundle\") pod \"654a9dfa-aba3-458e-87f4-bd73305627fe\" (UID: \"654a9dfa-aba3-458e-87f4-bd73305627fe\") "
Oct 30 00:08:51.598680 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:51.598601    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654a9dfa-aba3-458e-87f4-bd73305627fe-kube-api-access-pg6ct\" (UniqueName: \"kubernetes.io/projected/654a9dfa-aba3-458e-87f4-bd73305627fe-kube-api-access-pg6ct\") pod \"654a9dfa-aba3-458e-87f4-bd73305627fe\" (UID: \"654a9dfa-aba3-458e-87f4-bd73305627fe\") "
Oct 30 00:08:51.598680 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:51.598660    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-util\" (UniqueName: \"kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-util\") pod \"654a9dfa-aba3-458e-87f4-bd73305627fe\" (UID: \"654a9dfa-aba3-458e-87f4-bd73305627fe\") "
Oct 30 00:08:51.599129 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:51.599103    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-bundle" pod "654a9dfa-aba3-458e-87f4-bd73305627fe" (UID: "654a9dfa-aba3-458e-87f4-bd73305627fe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:08:51.600723 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:51.600701    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654a9dfa-aba3-458e-87f4-bd73305627fe-kube-api-access-pg6ct" pod "654a9dfa-aba3-458e-87f4-bd73305627fe" (UID: "654a9dfa-aba3-458e-87f4-bd73305627fe"). InnerVolumeSpecName "kube-api-access-pg6ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:08:51.602727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:51.602708    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-util" pod "654a9dfa-aba3-458e-87f4-bd73305627fe" (UID: "654a9dfa-aba3-458e-87f4-bd73305627fe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:08:51.699107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:51.699083    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg6ct\" (UniqueName: \"kubernetes.io/projected/654a9dfa-aba3-458e-87f4-bd73305627fe-kube-api-access-pg6ct\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:08:51.699107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:51.699106    4441 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-util\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:08:51.699282 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:51.699119    4441 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/654a9dfa-aba3-458e-87f4-bd73305627fe-bundle\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:08:52.461339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:52.461305    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh" event={"ID":"654a9dfa-aba3-458e-87f4-bd73305627fe","Type":"ContainerDied","Data":"b49545db95675c57ef13b67a3636c8d9d0448a707836d400ef5269efc18734d1"}
Oct 30 00:08:52.461482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:52.461351    4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b49545db95675c57ef13b67a3636c8d9d0448a707836d400ef5269efc18734d1"
Oct 30 00:08:52.461482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:52.461350    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716zhbh"
Oct 30 00:08:58.477404 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.477376    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"]
Oct 30 00:08:58.477770 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.477625    4441 cpu_manager.go:414] "containerMap: removing container" podUID="654a9dfa-aba3-458e-87f4-bd73305627fe" containerName="pull"
Oct 30 00:08:58.477770 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.477635    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654a9dfa-aba3-458e-87f4-bd73305627fe" containerName="pull"
Oct 30 00:08:58.477770 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.477648    4441 cpu_manager.go:414] "containerMap: removing container" podUID="654a9dfa-aba3-458e-87f4-bd73305627fe" containerName="extract"
Oct 30 00:08:58.477770 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.477653    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654a9dfa-aba3-458e-87f4-bd73305627fe" containerName="extract"
Oct 30 00:08:58.477770 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.477670    4441 cpu_manager.go:414] "containerMap: removing container" podUID="654a9dfa-aba3-458e-87f4-bd73305627fe" containerName="util"
Oct 30 00:08:58.477770 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.477675    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654a9dfa-aba3-458e-87f4-bd73305627fe" containerName="util"
Oct 30 00:08:58.477770 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.477710    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="654a9dfa-aba3-458e-87f4-bd73305627fe" containerName="extract"
Oct 30 00:08:58.479752 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.479736    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.498816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.498796    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 00:08:58.498931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.498813    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 00:08:58.506353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.506331    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 00:08:58.506458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.506403    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 00:08:58.506498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.506453    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 00:08:58.506620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.506606    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-84fdv\""
Oct 30 00:08:58.508382 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.508361    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"]
Oct 30 00:08:58.548838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.548813    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b434fab-1e04-434c-9574-274c0ba1edae-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.548964 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.548848    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b434fab-1e04-434c-9574-274c0ba1edae-tmp\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.548964 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.548869    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b434fab-1e04-434c-9574-274c0ba1edae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.549082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.549017    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b434fab-1e04-434c-9574-274c0ba1edae-manifests\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.549124 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.549089    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b434fab-1e04-434c-9574-274c0ba1edae-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.549124 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.549119    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzczp\" (UniqueName: \"kubernetes.io/projected/9b434fab-1e04-434c-9574-274c0ba1edae-kube-api-access-kzczp\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.649707 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.649685    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b434fab-1e04-434c-9574-274c0ba1edae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.649793 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.649716    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b434fab-1e04-434c-9574-274c0ba1edae-manifests\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.649793 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.649738    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b434fab-1e04-434c-9574-274c0ba1edae-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.649793 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.649761    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzczp\" (UniqueName: \"kubernetes.io/projected/9b434fab-1e04-434c-9574-274c0ba1edae-kube-api-access-kzczp\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.649941 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.649817    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b434fab-1e04-434c-9574-274c0ba1edae-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.649941 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.649858    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b434fab-1e04-434c-9574-274c0ba1edae-tmp\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.650234 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.650204    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b434fab-1e04-434c-9574-274c0ba1edae-manifests\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.652232 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.652201    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b434fab-1e04-434c-9574-274c0ba1edae-tmp\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.652351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.652316    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b434fab-1e04-434c-9574-274c0ba1edae-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.652401 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.652375    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b434fab-1e04-434c-9574-274c0ba1edae-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.652435 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.652416    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b434fab-1e04-434c-9574-274c0ba1edae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.672218 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.672197    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzczp\" (UniqueName: \"kubernetes.io/projected/9b434fab-1e04-434c-9574-274c0ba1edae-kube-api-access-kzczp\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m\" (UID: \"9b434fab-1e04-434c-9574-274c0ba1edae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:58.791645 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:58.791592    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:08:59.145559 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:59.145393    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"]
Oct 30 00:08:59.148433 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:08:59.148406    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b434fab_1e04_434c_9574_274c0ba1edae.slice/crio-35b10afd09bc5d821a519d024d409a51dec407580e9101eaad8d7868f2abfe00 WatchSource:0}: Error finding container 35b10afd09bc5d821a519d024d409a51dec407580e9101eaad8d7868f2abfe00: Status 404 returned error can't find the container with id 35b10afd09bc5d821a519d024d409a51dec407580e9101eaad8d7868f2abfe00
Oct 30 00:08:59.483000 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:08:59.482970    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m" event={"ID":"9b434fab-1e04-434c-9574-274c0ba1edae","Type":"ContainerStarted","Data":"35b10afd09bc5d821a519d024d409a51dec407580e9101eaad8d7868f2abfe00"}
Oct 30 00:09:02.498915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:02.498881    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m" event={"ID":"9b434fab-1e04-434c-9574-274c0ba1edae","Type":"ContainerStarted","Data":"ac6192335e81de2cf1dde56bea4f2fa5d806bc655953e2d923ddb440a90c77bc"}
Oct 30 00:09:03.503459 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:03.503415    4441 generic.go:356] "Generic (PLEG): container finished" podID="9b434fab-1e04-434c-9574-274c0ba1edae" containerID="ac6192335e81de2cf1dde56bea4f2fa5d806bc655953e2d923ddb440a90c77bc" exitCode=0
Oct 30 00:09:03.503838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:03.503471    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m" event={"ID":"9b434fab-1e04-434c-9574-274c0ba1edae","Type":"ContainerDied","Data":"ac6192335e81de2cf1dde56bea4f2fa5d806bc655953e2d923ddb440a90c77bc"}
Oct 30 00:09:03.897374 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:09:03.897299    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:09:04.508800 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:04.508758    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m" event={"ID":"9b434fab-1e04-434c-9574-274c0ba1edae","Type":"ContainerStarted","Data":"40658cd3d8d5fa48a804f5c6c4c1b0a7781416aff3b7d5989ad85300df49d422"}
Oct 30 00:09:04.509298 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:04.508898    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:09:04.617179 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:04.617125    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m" podStartSLOduration=2.903278163 podStartE2EDuration="6.617111856s" totalImagesPullingTime="3.221853417s" totalInitContainerRuntime="491.980276ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:08:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:04.605151074 +0000 UTC m=+254.130444951" watchObservedRunningTime="2026-10-30 00:09:04.617111856 +0000 UTC m=+254.142405733"
Oct 30 00:09:15.513644 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:15.513616    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m"
Oct 30 00:09:29.261203 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.261171    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5"]
Oct 30 00:09:29.264015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.263996    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5"
Oct 30 00:09:29.280553 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.280533    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 00:09:29.293280 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.293254    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5"]
Oct 30 00:09:29.296468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.296450    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-86lmt\""
Oct 30 00:09:29.366972 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.366942    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf5ks\" (UniqueName: \"kubernetes.io/projected/d5deb82b-2372-40b6-b196-3f3f90161ae0-kube-api-access-qf5ks\") pod \"cert-manager-webhook-5dbbbf75cc-7c9h5\" (UID: \"d5deb82b-2372-40b6-b196-3f3f90161ae0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5"
Oct 30 00:09:29.367102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.366998    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5deb82b-2372-40b6-b196-3f3f90161ae0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-7c9h5\" (UID: \"d5deb82b-2372-40b6-b196-3f3f90161ae0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5"
Oct 30 00:09:29.382489 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.382472    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 00:09:29.467931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.467890    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5deb82b-2372-40b6-b196-3f3f90161ae0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-7c9h5\" (UID: \"d5deb82b-2372-40b6-b196-3f3f90161ae0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5"
Oct 30 00:09:29.468059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.467956    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf5ks\" (UniqueName: \"kubernetes.io/projected/d5deb82b-2372-40b6-b196-3f3f90161ae0-kube-api-access-qf5ks\") pod \"cert-manager-webhook-5dbbbf75cc-7c9h5\" (UID: \"d5deb82b-2372-40b6-b196-3f3f90161ae0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5"
Oct 30 00:09:29.496420 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.496400    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5deb82b-2372-40b6-b196-3f3f90161ae0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-7c9h5\" (UID: \"d5deb82b-2372-40b6-b196-3f3f90161ae0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5"
Oct 30 00:09:29.500112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.500089    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf5ks\" (UniqueName: \"kubernetes.io/projected/d5deb82b-2372-40b6-b196-3f3f90161ae0-kube-api-access-qf5ks\") pod \"cert-manager-webhook-5dbbbf75cc-7c9h5\" (UID: \"d5deb82b-2372-40b6-b196-3f3f90161ae0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5"
Oct 30 00:09:29.584507 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.584445    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5"
Oct 30 00:09:29.874543 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:29.874438    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5"]
Oct 30 00:09:29.877218 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:09:29.877188    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5deb82b_2372_40b6_b196_3f3f90161ae0.slice/crio-6cdc3d2f4bcd8d1e6a7d5680795d66a813fcdd943a823c8832fab5485e816624 WatchSource:0}: Error finding container 6cdc3d2f4bcd8d1e6a7d5680795d66a813fcdd943a823c8832fab5485e816624: Status 404 returned error can't find the container with id 6cdc3d2f4bcd8d1e6a7d5680795d66a813fcdd943a823c8832fab5485e816624
Oct 30 00:09:30.595744 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:30.595708    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5" event={"ID":"d5deb82b-2372-40b6-b196-3f3f90161ae0","Type":"ContainerStarted","Data":"6cdc3d2f4bcd8d1e6a7d5680795d66a813fcdd943a823c8832fab5485e816624"}
Oct 30 00:09:33.612946 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:33.612911    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5" event={"ID":"d5deb82b-2372-40b6-b196-3f3f90161ae0","Type":"ContainerStarted","Data":"36385225ffd4b909c6fa27feed45f9b1650f833d5e1270807b71ffb3a171138b"}
Oct 30 00:09:33.613375 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:33.613058    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5"
Oct 30 00:09:33.652071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:33.652004    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5" podStartSLOduration=1.815926913 podStartE2EDuration="4.651988768s" totalImagesPullingTime="2.836061855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:09:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:33.651804045 +0000 UTC m=+283.177097922" watchObservedRunningTime="2026-10-30 00:09:33.651988768 +0000 UTC m=+283.177282648"
Oct 30 00:09:33.896197 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:09:33.896126    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:09:36.745413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:36.745380    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55"]
Oct 30 00:09:36.752177 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:36.752154    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55"
Oct 30 00:09:36.766095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:36.766076    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55"]
Oct 30 00:09:36.776536 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:36.776517    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-grd2l\""
Oct 30 00:09:36.832395 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:36.832375    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ws2\" (UniqueName: \"kubernetes.io/projected/890bac17-f789-4866-b0b1-7ba9f66c997b-kube-api-access-d9ws2\") pod \"cert-manager-cainjector-5c57bbd4fc-c9h55\" (UID: \"890bac17-f789-4866-b0b1-7ba9f66c997b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55"
Oct 30 00:09:36.832493 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:36.832421    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/890bac17-f789-4866-b0b1-7ba9f66c997b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-c9h55\" (UID: \"890bac17-f789-4866-b0b1-7ba9f66c997b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55"
Oct 30 00:09:36.933404 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:36.933379    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ws2\" (UniqueName: \"kubernetes.io/projected/890bac17-f789-4866-b0b1-7ba9f66c997b-kube-api-access-d9ws2\") pod \"cert-manager-cainjector-5c57bbd4fc-c9h55\" (UID: \"890bac17-f789-4866-b0b1-7ba9f66c997b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55"
Oct 30 00:09:36.933485 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:36.933422    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/890bac17-f789-4866-b0b1-7ba9f66c997b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-c9h55\" (UID: \"890bac17-f789-4866-b0b1-7ba9f66c997b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55"
Oct 30 00:09:36.955524 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:36.955507    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/890bac17-f789-4866-b0b1-7ba9f66c997b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-c9h55\" (UID: \"890bac17-f789-4866-b0b1-7ba9f66c997b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55"
Oct 30 00:09:36.955609 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:36.955594    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ws2\" (UniqueName: \"kubernetes.io/projected/890bac17-f789-4866-b0b1-7ba9f66c997b-kube-api-access-d9ws2\") pod \"cert-manager-cainjector-5c57bbd4fc-c9h55\" (UID: \"890bac17-f789-4866-b0b1-7ba9f66c997b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55"
Oct 30 00:09:37.062426 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:37.062373    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55"
Oct 30 00:09:37.354435 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:37.354360    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55"]
Oct 30 00:09:37.358571 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:09:37.358538    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod890bac17_f789_4866_b0b1_7ba9f66c997b.slice/crio-60d0091049452b4fe455b80c04658982fdda32b1f11b0f160230b29f9ed38425 WatchSource:0}: Error finding container 60d0091049452b4fe455b80c04658982fdda32b1f11b0f160230b29f9ed38425: Status 404 returned error can't find the container with id 60d0091049452b4fe455b80c04658982fdda32b1f11b0f160230b29f9ed38425
Oct 30 00:09:37.628498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:37.628423    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55" event={"ID":"890bac17-f789-4866-b0b1-7ba9f66c997b","Type":"ContainerStarted","Data":"d0510678186aea24708e61a2d1a074501902811e4e03b37cfd8f785744b49546"}
Oct 30 00:09:37.628498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:37.628461    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55" event={"ID":"890bac17-f789-4866-b0b1-7ba9f66c997b","Type":"ContainerStarted","Data":"60d0091049452b4fe455b80c04658982fdda32b1f11b0f160230b29f9ed38425"}
Oct 30 00:09:37.666884 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:37.666837    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c9h55" podStartSLOduration=1.666825067 podStartE2EDuration="1.666825067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:37.666013858 +0000 UTC m=+287.191307737" watchObservedRunningTime="2026-10-30 00:09:37.666825067 +0000 UTC m=+287.192118943"
Oct 30 00:09:39.618439 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:39.618407    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5"
Oct 30 00:09:50.059406 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.059373    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-t2mxs"]
Oct 30 00:09:50.061706 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.061686    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t2mxs"
Oct 30 00:09:50.082820 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.082796    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 00:09:50.094118 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.094098    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 00:09:50.095311 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.095289    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-t2mxs"]
Oct 30 00:09:50.105366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.105351    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-trlh8\""
Oct 30 00:09:50.235434 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.235409    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2cnr\" (UniqueName: \"kubernetes.io/projected/03e20d41-ae7d-4ea8-8528-12f00023b3b4-kube-api-access-l2cnr\") pod \"authorino-operator-7b7bb4bc77-t2mxs\" (UID: \"03e20d41-ae7d-4ea8-8528-12f00023b3b4\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-t2mxs"
Oct 30 00:09:50.336303 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.336234    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2cnr\" (UniqueName: \"kubernetes.io/projected/03e20d41-ae7d-4ea8-8528-12f00023b3b4-kube-api-access-l2cnr\") pod \"authorino-operator-7b7bb4bc77-t2mxs\" (UID: \"03e20d41-ae7d-4ea8-8528-12f00023b3b4\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-t2mxs"
Oct 30 00:09:50.361095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.361072    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2cnr\" (UniqueName: \"kubernetes.io/projected/03e20d41-ae7d-4ea8-8528-12f00023b3b4-kube-api-access-l2cnr\") pod \"authorino-operator-7b7bb4bc77-t2mxs\" (UID: \"03e20d41-ae7d-4ea8-8528-12f00023b3b4\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-t2mxs"
Oct 30 00:09:50.372862 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.372841    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t2mxs"
Oct 30 00:09:50.614805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.614737    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:09:50.614958 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.614807    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:09:50.663890 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.663803    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-t2mxs"]
Oct 30 00:09:50.666515 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:09:50.666491    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e20d41_ae7d_4ea8_8528_12f00023b3b4.slice/crio-878d7c835e035668a7edbdeee2b5f80be81b2c09637e6dd79c3f374a5aeeca6e WatchSource:0}: Error finding container 878d7c835e035668a7edbdeee2b5f80be81b2c09637e6dd79c3f374a5aeeca6e: Status 404 returned error can't find the container with id 878d7c835e035668a7edbdeee2b5f80be81b2c09637e6dd79c3f374a5aeeca6e
Oct 30 00:09:50.676202 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:50.676176    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t2mxs" event={"ID":"03e20d41-ae7d-4ea8-8528-12f00023b3b4","Type":"ContainerStarted","Data":"878d7c835e035668a7edbdeee2b5f80be81b2c09637e6dd79c3f374a5aeeca6e"}
Oct 30 00:09:51.620107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.620073    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"]
Oct 30 00:09:51.623007 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.622989    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"
Oct 30 00:09:51.637267 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.637246    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 00:09:51.637379 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.637286    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 00:09:51.637379 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.637333    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 00:09:51.637498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.637403    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 00:09:51.648674 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.648651    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6bbvl\""
Oct 30 00:09:51.650352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.650327    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"]
Oct 30 00:09:51.746882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.746842    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ed0996-1da5-471c-a110-fb73de534126-tls-certs\") pod \"kube-auth-proxy-64d88dd7bf-bblqz\" (UID: \"36ed0996-1da5-471c-a110-fb73de534126\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"
Oct 30 00:09:51.747080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.746931    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36ed0996-1da5-471c-a110-fb73de534126-tmp\") pod \"kube-auth-proxy-64d88dd7bf-bblqz\" (UID: \"36ed0996-1da5-471c-a110-fb73de534126\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"
Oct 30 00:09:51.747418 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.747389    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mntw9\" (UniqueName: \"kubernetes.io/projected/36ed0996-1da5-471c-a110-fb73de534126-kube-api-access-mntw9\") pod \"kube-auth-proxy-64d88dd7bf-bblqz\" (UID: \"36ed0996-1da5-471c-a110-fb73de534126\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"
Oct 30 00:09:51.848375 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.848343    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36ed0996-1da5-471c-a110-fb73de534126-tmp\") pod \"kube-auth-proxy-64d88dd7bf-bblqz\" (UID: \"36ed0996-1da5-471c-a110-fb73de534126\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"
Oct 30 00:09:51.848539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.848436    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mntw9\" (UniqueName: \"kubernetes.io/projected/36ed0996-1da5-471c-a110-fb73de534126-kube-api-access-mntw9\") pod \"kube-auth-proxy-64d88dd7bf-bblqz\" (UID: \"36ed0996-1da5-471c-a110-fb73de534126\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"
Oct 30 00:09:51.848539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.848492    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ed0996-1da5-471c-a110-fb73de534126-tls-certs\") pod \"kube-auth-proxy-64d88dd7bf-bblqz\" (UID: \"36ed0996-1da5-471c-a110-fb73de534126\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"
Oct 30 00:09:51.850898 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.850872    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36ed0996-1da5-471c-a110-fb73de534126-tmp\") pod \"kube-auth-proxy-64d88dd7bf-bblqz\" (UID: \"36ed0996-1da5-471c-a110-fb73de534126\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"
Oct 30 00:09:51.851275 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.851252    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ed0996-1da5-471c-a110-fb73de534126-tls-certs\") pod \"kube-auth-proxy-64d88dd7bf-bblqz\" (UID: \"36ed0996-1da5-471c-a110-fb73de534126\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"
Oct 30 00:09:51.876927 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.876862    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mntw9\" (UniqueName: \"kubernetes.io/projected/36ed0996-1da5-471c-a110-fb73de534126-kube-api-access-mntw9\") pod \"kube-auth-proxy-64d88dd7bf-bblqz\" (UID: \"36ed0996-1da5-471c-a110-fb73de534126\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"
Oct 30 00:09:51.936874 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:51.936849    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"
Oct 30 00:09:52.243574 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:52.243470    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz"]
Oct 30 00:09:52.250399 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:09:52.250348    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ed0996_1da5_471c_a110_fb73de534126.slice/crio-d5491a77e8ffda97359fc6322a975bb6c29879f9fc2c43a9b2864605f2b12688 WatchSource:0}: Error finding container d5491a77e8ffda97359fc6322a975bb6c29879f9fc2c43a9b2864605f2b12688: Status 404 returned error can't find the container with id d5491a77e8ffda97359fc6322a975bb6c29879f9fc2c43a9b2864605f2b12688
Oct 30 00:09:52.686696 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:52.686612    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz" event={"ID":"36ed0996-1da5-471c-a110-fb73de534126","Type":"ContainerStarted","Data":"d5491a77e8ffda97359fc6322a975bb6c29879f9fc2c43a9b2864605f2b12688"}
Oct 30 00:09:53.693440 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:53.693408    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t2mxs" event={"ID":"03e20d41-ae7d-4ea8-8528-12f00023b3b4","Type":"ContainerStarted","Data":"9783d54d1636d2c623935e5e84ec312fcdaf21d508eef1e2cad92534a13d3743"}
Oct 30 00:09:53.693912 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:53.693486    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t2mxs"
Oct 30 00:09:54.698675 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:54.698627    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz" event={"ID":"36ed0996-1da5-471c-a110-fb73de534126","Type":"ContainerStarted","Data":"5e4885b20c6e0f8eb8e3dab6d2b2f0c7938292bce3a12d3871ba8ff589cd20ec"}
Oct 30 00:09:54.738629 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:54.738583    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t2mxs" podStartSLOduration=2.382217483 podStartE2EDuration="4.738570773s" totalImagesPullingTime="2.35635329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:53.73904889 +0000 UTC m=+303.264342767" watchObservedRunningTime="2026-10-30 00:09:54.738570773 +0000 UTC m=+304.263864649"
Oct 30 00:09:54.738765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:09:54.738684    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-bblqz" podStartSLOduration=1.414564446 podStartE2EDuration="3.738675161s" totalImagesPullingTime="2.324110715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:54.736653297 +0000 UTC m=+304.261947174" watchObservedRunningTime="2026-10-30 00:09:54.738675161 +0000 UTC m=+304.263969039"
Oct 30 00:10:00.281312 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:00.281281    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"]
Oct 30 00:10:00.283836 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:00.283818    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"
Oct 30 00:10:00.298672 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:00.298650    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"]
Oct 30 00:10:00.309010 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:00.308983    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wl68b\""
Oct 30 00:10:00.316249 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:00.316225    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/185b11f8-5a8a-4282-b9ae-53c9182d0668-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tmkmd\" (UID: \"185b11f8-5a8a-4282-b9ae-53c9182d0668\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"
Oct 30 00:10:00.316326 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:00.316276    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvvfw\" (UniqueName: \"kubernetes.io/projected/185b11f8-5a8a-4282-b9ae-53c9182d0668-kube-api-access-mvvfw\") pod \"kuadrant-operator-controller-manager-678d87766f-tmkmd\" (UID: \"185b11f8-5a8a-4282-b9ae-53c9182d0668\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"
Oct 30 00:10:00.416716 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:00.416691    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/185b11f8-5a8a-4282-b9ae-53c9182d0668-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tmkmd\" (UID: \"185b11f8-5a8a-4282-b9ae-53c9182d0668\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"
Oct 30 00:10:00.416810 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:00.416725    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvvfw\" (UniqueName: \"kubernetes.io/projected/185b11f8-5a8a-4282-b9ae-53c9182d0668-kube-api-access-mvvfw\") pod \"kuadrant-operator-controller-manager-678d87766f-tmkmd\" (UID: \"185b11f8-5a8a-4282-b9ae-53c9182d0668\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"
Oct 30 00:10:00.417062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:00.417045    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/185b11f8-5a8a-4282-b9ae-53c9182d0668-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tmkmd\" (UID: \"185b11f8-5a8a-4282-b9ae-53c9182d0668\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"
Oct 30 00:10:00.468619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:00.468589    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvvfw\" (UniqueName: \"kubernetes.io/projected/185b11f8-5a8a-4282-b9ae-53c9182d0668-kube-api-access-mvvfw\") pod \"kuadrant-operator-controller-manager-678d87766f-tmkmd\" (UID: \"185b11f8-5a8a-4282-b9ae-53c9182d0668\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"
Oct 30 00:10:00.594712 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:00.594664    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"
Oct 30 00:10:00.912764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:00.912724    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"]
Oct 30 00:10:00.915715 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:10:00.915688    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185b11f8_5a8a_4282_b9ae_53c9182d0668.slice/crio-3cddeb175fc96406af760c2883b96b6fcbbfed9eb1fb61d1e9707d0e5ef1ac16 WatchSource:0}: Error finding container 3cddeb175fc96406af760c2883b96b6fcbbfed9eb1fb61d1e9707d0e5ef1ac16: Status 404 returned error can't find the container with id 3cddeb175fc96406af760c2883b96b6fcbbfed9eb1fb61d1e9707d0e5ef1ac16
Oct 30 00:10:00.918144 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:00.918125    4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:10:01.725253 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:01.725218    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd" event={"ID":"185b11f8-5a8a-4282-b9ae-53c9182d0668","Type":"ContainerStarted","Data":"3cddeb175fc96406af760c2883b96b6fcbbfed9eb1fb61d1e9707d0e5ef1ac16"}
Oct 30 00:10:03.901977 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:10:03.901940    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:10:04.700602 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:04.700571    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t2mxs"
Oct 30 00:10:06.745445 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:06.745409    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd" event={"ID":"185b11f8-5a8a-4282-b9ae-53c9182d0668","Type":"ContainerStarted","Data":"14d881485fc61f9853d49e2b3997828bb05ba133934d3ee95ce1b6f4affe5f1c"}
Oct 30 00:10:06.745853 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:06.745530    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"
Oct 30 00:10:06.810663 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:06.810614    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd" podStartSLOduration=1.62794646 podStartE2EDuration="6.810601984s" totalImagesPullingTime="5.182655524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:10:06.810291114 +0000 UTC m=+316.335584991" watchObservedRunningTime="2026-10-30 00:10:06.810601984 +0000 UTC m=+316.335895860"
Oct 30 00:10:17.752162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:17.752130    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"
Oct 30 00:10:33.898456 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:10:33.898415    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:10:39.083224 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.083194    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-t8lbz"]
Oct 30 00:10:39.085677 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.085652    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-t8lbz"
Oct 30 00:10:39.086126 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.086103    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:10:39.088681 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.088658    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"
Oct 30 00:10:39.102297 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.102272    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-t8lbz"]
Oct 30 00:10:39.112603 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.112581    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6wl6d\""
Oct 30 00:10:39.112732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.112597    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hsh2c\""
Oct 30 00:10:39.112806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.112786    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 00:10:39.113984 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.113966    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:10:39.181383 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.181351    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:10:39.213794 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.213760    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jr45\" (UniqueName: \"kubernetes.io/projected/89f3f81c-652b-47c3-9b2b-71c389d43e10-kube-api-access-5jr45\") pod \"authorino-56c6c9ddd5-t8lbz\" (UID: \"89f3f81c-652b-47c3-9b2b-71c389d43e10\") " pod="kuadrant-system/authorino-56c6c9ddd5-t8lbz"
Oct 30 00:10:39.213794 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.213796    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm4pc\" (UniqueName: \"kubernetes.io/projected/7883212e-1f9e-471c-be5d-51d0f0b20188-kube-api-access-tm4pc\") pod \"limitador-limitador-7c9b87d574-j8jsw\" (UID: \"7883212e-1f9e-471c-be5d-51d0f0b20188\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"
Oct 30 00:10:39.214006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.213826    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7883212e-1f9e-471c-be5d-51d0f0b20188-config-file\") pod \"limitador-limitador-7c9b87d574-j8jsw\" (UID: \"7883212e-1f9e-471c-be5d-51d0f0b20188\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"
Oct 30 00:10:39.314702 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.314665    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jr45\" (UniqueName: \"kubernetes.io/projected/89f3f81c-652b-47c3-9b2b-71c389d43e10-kube-api-access-5jr45\") pod \"authorino-56c6c9ddd5-t8lbz\" (UID: \"89f3f81c-652b-47c3-9b2b-71c389d43e10\") " pod="kuadrant-system/authorino-56c6c9ddd5-t8lbz"
Oct 30 00:10:39.314849 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.314708    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm4pc\" (UniqueName: \"kubernetes.io/projected/7883212e-1f9e-471c-be5d-51d0f0b20188-kube-api-access-tm4pc\") pod \"limitador-limitador-7c9b87d574-j8jsw\" (UID: \"7883212e-1f9e-471c-be5d-51d0f0b20188\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"
Oct 30 00:10:39.314849 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.314738    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7883212e-1f9e-471c-be5d-51d0f0b20188-config-file\") pod \"limitador-limitador-7c9b87d574-j8jsw\" (UID: \"7883212e-1f9e-471c-be5d-51d0f0b20188\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"
Oct 30 00:10:39.315393 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.315373    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7883212e-1f9e-471c-be5d-51d0f0b20188-config-file\") pod \"limitador-limitador-7c9b87d574-j8jsw\" (UID: \"7883212e-1f9e-471c-be5d-51d0f0b20188\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"
Oct 30 00:10:39.337223 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.337161    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jr45\" (UniqueName: \"kubernetes.io/projected/89f3f81c-652b-47c3-9b2b-71c389d43e10-kube-api-access-5jr45\") pod \"authorino-56c6c9ddd5-t8lbz\" (UID: \"89f3f81c-652b-47c3-9b2b-71c389d43e10\") " pod="kuadrant-system/authorino-56c6c9ddd5-t8lbz"
Oct 30 00:10:39.337335 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.337233    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm4pc\" (UniqueName: \"kubernetes.io/projected/7883212e-1f9e-471c-be5d-51d0f0b20188-kube-api-access-tm4pc\") pod \"limitador-limitador-7c9b87d574-j8jsw\" (UID: \"7883212e-1f9e-471c-be5d-51d0f0b20188\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"
Oct 30 00:10:39.397728 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.397699    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-t8lbz"
Oct 30 00:10:39.404790 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.404767    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"
Oct 30 00:10:39.695001 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.694978    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-t8lbz"]
Oct 30 00:10:39.696832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:10:39.696805    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89f3f81c_652b_47c3_9b2b_71c389d43e10.slice/crio-0f0c54649ea83f46f8422d5e4ce329a21f200fca6bec3fd64405832d3754d98b WatchSource:0}: Error finding container 0f0c54649ea83f46f8422d5e4ce329a21f200fca6bec3fd64405832d3754d98b: Status 404 returned error can't find the container with id 0f0c54649ea83f46f8422d5e4ce329a21f200fca6bec3fd64405832d3754d98b
Oct 30 00:10:39.707143 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.707056    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:10:39.709071 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:10:39.709046    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7883212e_1f9e_471c_be5d_51d0f0b20188.slice/crio-30130d6ea13e15fafac50e1fc24a878de92cfde51d3d40c2b0b8c505e8c837ab WatchSource:0}: Error finding container 30130d6ea13e15fafac50e1fc24a878de92cfde51d3d40c2b0b8c505e8c837ab: Status 404 returned error can't find the container with id 30130d6ea13e15fafac50e1fc24a878de92cfde51d3d40c2b0b8c505e8c837ab
Oct 30 00:10:39.877017 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.876984    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8jsw" event={"ID":"7883212e-1f9e-471c-be5d-51d0f0b20188","Type":"ContainerStarted","Data":"30130d6ea13e15fafac50e1fc24a878de92cfde51d3d40c2b0b8c505e8c837ab"}
Oct 30 00:10:39.878318 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:39.878296    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-t8lbz" event={"ID":"89f3f81c-652b-47c3-9b2b-71c389d43e10","Type":"ContainerStarted","Data":"0f0c54649ea83f46f8422d5e4ce329a21f200fca6bec3fd64405832d3754d98b"}
Oct 30 00:10:40.886411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:40.886308    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8jsw" event={"ID":"7883212e-1f9e-471c-be5d-51d0f0b20188","Type":"ContainerStarted","Data":"547c51d45043393628053eb66fd491d0f951a3f28d81b92f0347b9c6d32c8346"}
Oct 30 00:10:40.886813 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:40.886415    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"
Oct 30 00:10:40.926111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:40.926060    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8jsw" podStartSLOduration=1.082019461 podStartE2EDuration="1.926045869s" totalImagesPullingTime="844.026408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:10:40.924714842 +0000 UTC m=+350.450008720" watchObservedRunningTime="2026-10-30 00:10:40.926045869 +0000 UTC m=+350.451339749"
Oct 30 00:10:41.891158 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:41.891124    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-t8lbz" event={"ID":"89f3f81c-652b-47c3-9b2b-71c389d43e10","Type":"ContainerStarted","Data":"39d558261b43fae075ec8fe1901ea0998508e14b539ee592bd9486df04dc6a39"}
Oct 30 00:10:41.928430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:41.928381    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-t8lbz" podStartSLOduration=0.850196241 podStartE2EDuration="2.928366832s" totalImagesPullingTime="2.078170591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:10:41.927644445 +0000 UTC m=+351.452938322" watchObservedRunningTime="2026-10-30 00:10:41.928366832 +0000 UTC m=+351.453660758"
Oct 30 00:10:51.892678 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:51.892646    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"
Oct 30 00:10:53.821768 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:53.821733    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-877hw"]
Oct 30 00:10:53.824237 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:53.824218    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-877hw"
Oct 30 00:10:53.841319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:53.841301    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 00:10:53.843363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:53.843330    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-877hw"]
Oct 30 00:10:53.853368 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:53.853348    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 00:10:53.853450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:53.853369    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 00:10:53.853450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:53.853387    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qgq2x\""
Oct 30 00:10:53.940502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:53.940471    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88fc9fb9-507f-455a-93b1-6d158cc5896f-cert\") pod \"kserve-controller-manager-776cc66d47-877hw\" (UID: \"88fc9fb9-507f-455a-93b1-6d158cc5896f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-877hw"
Oct 30 00:10:53.940673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:53.940550    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q47qq\" (UniqueName: \"kubernetes.io/projected/88fc9fb9-507f-455a-93b1-6d158cc5896f-kube-api-access-q47qq\") pod \"kserve-controller-manager-776cc66d47-877hw\" (UID: \"88fc9fb9-507f-455a-93b1-6d158cc5896f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-877hw"
Oct 30 00:10:53.951887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:53.951835    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-9lbf8"]
Oct 30 00:10:53.954365 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:53.954345    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-9lbf8"
Oct 30 00:10:53.973463 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:53.973445    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-9lbf8"]
Oct 30 00:10:54.040582 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.040560    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-hdvbq\""
Oct 30 00:10:54.040720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.040617    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 00:10:54.040720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.040669    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 00:10:54.040995 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.040971    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q47qq\" (UniqueName: \"kubernetes.io/projected/88fc9fb9-507f-455a-93b1-6d158cc5896f-kube-api-access-q47qq\") pod \"kserve-controller-manager-776cc66d47-877hw\" (UID: \"88fc9fb9-507f-455a-93b1-6d158cc5896f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-877hw"
Oct 30 00:10:54.041110 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.041012    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ba56bd4-0351-41df-9717-c0ba2ec2d124-metrics-certs\") pod \"odh-model-controller-769875db56-9lbf8\" (UID: \"5ba56bd4-0351-41df-9717-c0ba2ec2d124\") " pod="opendatahub/odh-model-controller-769875db56-9lbf8"
Oct 30 00:10:54.041110 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.041088    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88fc9fb9-507f-455a-93b1-6d158cc5896f-cert\") pod \"kserve-controller-manager-776cc66d47-877hw\" (UID: \"88fc9fb9-507f-455a-93b1-6d158cc5896f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-877hw"
Oct 30 00:10:54.041195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.041155    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ba56bd4-0351-41df-9717-c0ba2ec2d124-cert\") pod \"odh-model-controller-769875db56-9lbf8\" (UID: \"5ba56bd4-0351-41df-9717-c0ba2ec2d124\") " pod="opendatahub/odh-model-controller-769875db56-9lbf8"
Oct 30 00:10:54.041244 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.041195    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcxf7\" (UniqueName: \"kubernetes.io/projected/5ba56bd4-0351-41df-9717-c0ba2ec2d124-kube-api-access-hcxf7\") pod \"odh-model-controller-769875db56-9lbf8\" (UID: \"5ba56bd4-0351-41df-9717-c0ba2ec2d124\") " pod="opendatahub/odh-model-controller-769875db56-9lbf8"
Oct 30 00:10:54.043532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.043511    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88fc9fb9-507f-455a-93b1-6d158cc5896f-cert\") pod \"kserve-controller-manager-776cc66d47-877hw\" (UID: \"88fc9fb9-507f-455a-93b1-6d158cc5896f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-877hw"
Oct 30 00:10:54.063385 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.063355    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q47qq\" (UniqueName: \"kubernetes.io/projected/88fc9fb9-507f-455a-93b1-6d158cc5896f-kube-api-access-q47qq\") pod \"kserve-controller-manager-776cc66d47-877hw\" (UID: \"88fc9fb9-507f-455a-93b1-6d158cc5896f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-877hw"
Oct 30 00:10:54.135829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.135767    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-877hw"
Oct 30 00:10:54.142249 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.142229    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ba56bd4-0351-41df-9717-c0ba2ec2d124-metrics-certs\") pod \"odh-model-controller-769875db56-9lbf8\" (UID: \"5ba56bd4-0351-41df-9717-c0ba2ec2d124\") " pod="opendatahub/odh-model-controller-769875db56-9lbf8"
Oct 30 00:10:54.142354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.142291    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ba56bd4-0351-41df-9717-c0ba2ec2d124-cert\") pod \"odh-model-controller-769875db56-9lbf8\" (UID: \"5ba56bd4-0351-41df-9717-c0ba2ec2d124\") " pod="opendatahub/odh-model-controller-769875db56-9lbf8"
Oct 30 00:10:54.142354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.142321    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcxf7\" (UniqueName: \"kubernetes.io/projected/5ba56bd4-0351-41df-9717-c0ba2ec2d124-kube-api-access-hcxf7\") pod \"odh-model-controller-769875db56-9lbf8\" (UID: \"5ba56bd4-0351-41df-9717-c0ba2ec2d124\") " pod="opendatahub/odh-model-controller-769875db56-9lbf8"
Oct 30 00:10:54.144677 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.144657    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ba56bd4-0351-41df-9717-c0ba2ec2d124-metrics-certs\") pod \"odh-model-controller-769875db56-9lbf8\" (UID: \"5ba56bd4-0351-41df-9717-c0ba2ec2d124\") " pod="opendatahub/odh-model-controller-769875db56-9lbf8"
Oct 30 00:10:54.144756 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.144710    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ba56bd4-0351-41df-9717-c0ba2ec2d124-cert\") pod \"odh-model-controller-769875db56-9lbf8\" (UID: \"5ba56bd4-0351-41df-9717-c0ba2ec2d124\") " pod="opendatahub/odh-model-controller-769875db56-9lbf8"
Oct 30 00:10:54.167424 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.167398    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcxf7\" (UniqueName: \"kubernetes.io/projected/5ba56bd4-0351-41df-9717-c0ba2ec2d124-kube-api-access-hcxf7\") pod \"odh-model-controller-769875db56-9lbf8\" (UID: \"5ba56bd4-0351-41df-9717-c0ba2ec2d124\") " pod="opendatahub/odh-model-controller-769875db56-9lbf8"
Oct 30 00:10:54.265622 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.265586    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-9lbf8"
Oct 30 00:10:54.426701 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.426644    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-877hw"]
Oct 30 00:10:54.428879 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:10:54.428851    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88fc9fb9_507f_455a_93b1_6d158cc5896f.slice/crio-189d3ebc95f45f603bf9d5def363da39aa45cdd7d8197a502c03c2e29995d2fb WatchSource:0}: Error finding container 189d3ebc95f45f603bf9d5def363da39aa45cdd7d8197a502c03c2e29995d2fb: Status 404 returned error can't find the container with id 189d3ebc95f45f603bf9d5def363da39aa45cdd7d8197a502c03c2e29995d2fb
Oct 30 00:10:54.554505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.554473    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-9lbf8"]
Oct 30 00:10:54.556462 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:10:54.556438    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba56bd4_0351_41df_9717_c0ba2ec2d124.slice/crio-28dfe5cf78e3097841540a0e0028e5468bcc544afa3b8c5d957aa36aa6ebb282 WatchSource:0}: Error finding container 28dfe5cf78e3097841540a0e0028e5468bcc544afa3b8c5d957aa36aa6ebb282: Status 404 returned error can't find the container with id 28dfe5cf78e3097841540a0e0028e5468bcc544afa3b8c5d957aa36aa6ebb282
Oct 30 00:10:54.942755 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.942714    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-9lbf8" event={"ID":"5ba56bd4-0351-41df-9717-c0ba2ec2d124","Type":"ContainerStarted","Data":"28dfe5cf78e3097841540a0e0028e5468bcc544afa3b8c5d957aa36aa6ebb282"}
Oct 30 00:10:54.943968 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:54.943924    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-877hw" event={"ID":"88fc9fb9-507f-455a-93b1-6d158cc5896f","Type":"ContainerStarted","Data":"189d3ebc95f45f603bf9d5def363da39aa45cdd7d8197a502c03c2e29995d2fb"}
Oct 30 00:10:56.956765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:56.956729    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-877hw" event={"ID":"88fc9fb9-507f-455a-93b1-6d158cc5896f","Type":"ContainerStarted","Data":"4f164873739d35d722624fae418237cd7367bcb2a002c2b37e828c17857d2e6f"}
Oct 30 00:10:56.957263 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:56.956829    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-877hw"
Oct 30 00:10:56.996118 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:56.995879    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-877hw" podStartSLOduration=1.690614487 podStartE2EDuration="3.995863843s" totalImagesPullingTime="2.305249356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:10:56.99538688 +0000 UTC m=+366.520680760" watchObservedRunningTime="2026-10-30 00:10:56.995863843 +0000 UTC m=+366.521157720"
Oct 30 00:10:57.962500 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:57.962456    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-9lbf8" event={"ID":"5ba56bd4-0351-41df-9717-c0ba2ec2d124","Type":"ContainerStarted","Data":"83e5b52f57d3717c90d8a3a8727cdcc85c22ba61b7bd2954d7248919ebcdf352"}
Oct 30 00:10:57.963005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:57.962572    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-9lbf8"
Oct 30 00:10:58.002992 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:10:58.002937    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-9lbf8" podStartSLOduration=2.128812745 podStartE2EDuration="5.002921008s" totalImagesPullingTime="2.874108263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:10:58.00123666 +0000 UTC m=+367.526530538" watchObservedRunningTime="2026-10-30 00:10:58.002921008 +0000 UTC m=+367.528214882"
Oct 30 00:11:03.898473 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:11:03.898422    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:11:08.967933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:11:08.967907    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-9lbf8"
Oct 30 00:11:27.968308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:11:27.968278    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-877hw"
Oct 30 00:11:33.897213 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:11:33.897175    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:11:52.642109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:11:52.642069    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:11:59.854912 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:11:59.854876    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-msdnc"]
Oct 30 00:11:59.857442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:11:59.857422    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-msdnc"
Oct 30 00:11:59.874561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:11:59.874535    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-msdnc"]
Oct 30 00:11:59.885510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:11:59.885491    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzlcv\" (UniqueName: \"kubernetes.io/projected/094be578-168a-44eb-b8bb-6dfceba40c5c-kube-api-access-jzlcv\") pod \"authorino-6f6944bcdd-msdnc\" (UID: \"094be578-168a-44eb-b8bb-6dfceba40c5c\") " pod="kuadrant-system/authorino-6f6944bcdd-msdnc"
Oct 30 00:11:59.986889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:11:59.986860    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlcv\" (UniqueName: \"kubernetes.io/projected/094be578-168a-44eb-b8bb-6dfceba40c5c-kube-api-access-jzlcv\") pod \"authorino-6f6944bcdd-msdnc\" (UID: \"094be578-168a-44eb-b8bb-6dfceba40c5c\") " pod="kuadrant-system/authorino-6f6944bcdd-msdnc"
Oct 30 00:12:00.014456 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.014427    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlcv\" (UniqueName: \"kubernetes.io/projected/094be578-168a-44eb-b8bb-6dfceba40c5c-kube-api-access-jzlcv\") pod \"authorino-6f6944bcdd-msdnc\" (UID: \"094be578-168a-44eb-b8bb-6dfceba40c5c\") " pod="kuadrant-system/authorino-6f6944bcdd-msdnc"
Oct 30 00:12:00.043717 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.043680    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-msdnc"]
Oct 30 00:12:00.043925 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.043903    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-msdnc"
Oct 30 00:12:00.069562 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.069528    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n6l7z"]
Oct 30 00:12:00.072399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.072377    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n6l7z"
Oct 30 00:12:00.088618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.088595    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 00:12:00.090532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.090316    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n6l7z"]
Oct 30 00:12:00.188797 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.188729    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-tls-cert\") pod \"authorino-6889cbd8c4-n6l7z\" (UID: \"91e13842-ea9d-48f9-bec1-e6ae0ad4351a\") " pod="kuadrant-system/authorino-6889cbd8c4-n6l7z"
Oct 30 00:12:00.188797 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.188763    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqz6n\" (UniqueName: \"kubernetes.io/projected/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-kube-api-access-lqz6n\") pod \"authorino-6889cbd8c4-n6l7z\" (UID: \"91e13842-ea9d-48f9-bec1-e6ae0ad4351a\") " pod="kuadrant-system/authorino-6889cbd8c4-n6l7z"
Oct 30 00:12:00.231558 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.231529    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n6l7z"]
Oct 30 00:12:00.231752 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:12:00.231726    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-lqz6n tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-n6l7z" podUID="91e13842-ea9d-48f9-bec1-e6ae0ad4351a"
Oct 30 00:12:00.258457 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.258426    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54c4767776-4jqcd"]
Oct 30 00:12:00.260811 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.260790    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54c4767776-4jqcd"
Oct 30 00:12:00.280567 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.280543    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54c4767776-4jqcd"]
Oct 30 00:12:00.289386 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.289364    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-tls-cert\") pod \"authorino-6889cbd8c4-n6l7z\" (UID: \"91e13842-ea9d-48f9-bec1-e6ae0ad4351a\") " pod="kuadrant-system/authorino-6889cbd8c4-n6l7z"
Oct 30 00:12:00.289479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.289397    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3db823ab-4341-4ed0-ad70-412edeb3aefa-tls-cert\") pod \"authorino-54c4767776-4jqcd\" (UID: \"3db823ab-4341-4ed0-ad70-412edeb3aefa\") " pod="kuadrant-system/authorino-54c4767776-4jqcd"
Oct 30 00:12:00.289479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.289419    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqz6n\" (UniqueName: \"kubernetes.io/projected/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-kube-api-access-lqz6n\") pod \"authorino-6889cbd8c4-n6l7z\" (UID: \"91e13842-ea9d-48f9-bec1-e6ae0ad4351a\") " pod="kuadrant-system/authorino-6889cbd8c4-n6l7z"
Oct 30 00:12:00.289671 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.289653    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvbcw\" (UniqueName: \"kubernetes.io/projected/3db823ab-4341-4ed0-ad70-412edeb3aefa-kube-api-access-jvbcw\") pod \"authorino-54c4767776-4jqcd\" (UID: \"3db823ab-4341-4ed0-ad70-412edeb3aefa\") " pod="kuadrant-system/authorino-54c4767776-4jqcd"
Oct 30 00:12:00.291782 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.291763    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-tls-cert\") pod \"authorino-6889cbd8c4-n6l7z\" (UID: \"91e13842-ea9d-48f9-bec1-e6ae0ad4351a\") " pod="kuadrant-system/authorino-6889cbd8c4-n6l7z"
Oct 30 00:12:00.312011 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.311986    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqz6n\" (UniqueName: \"kubernetes.io/projected/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-kube-api-access-lqz6n\") pod \"authorino-6889cbd8c4-n6l7z\" (UID: \"91e13842-ea9d-48f9-bec1-e6ae0ad4351a\") " pod="kuadrant-system/authorino-6889cbd8c4-n6l7z"
Oct 30 00:12:00.341090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.341071    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-msdnc"]
Oct 30 00:12:00.343177 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:12:00.343154    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod094be578_168a_44eb_b8bb_6dfceba40c5c.slice/crio-f5876ccd695188cfd0f5bcede4c6344d8c5a4e77f85e5848076e5230e171dbbc WatchSource:0}: Error finding container f5876ccd695188cfd0f5bcede4c6344d8c5a4e77f85e5848076e5230e171dbbc: Status 404 returned error can't find the container with id f5876ccd695188cfd0f5bcede4c6344d8c5a4e77f85e5848076e5230e171dbbc
Oct 30 00:12:00.390585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.390559    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvbcw\" (UniqueName: \"kubernetes.io/projected/3db823ab-4341-4ed0-ad70-412edeb3aefa-kube-api-access-jvbcw\") pod \"authorino-54c4767776-4jqcd\" (UID: \"3db823ab-4341-4ed0-ad70-412edeb3aefa\") " pod="kuadrant-system/authorino-54c4767776-4jqcd"
Oct 30 00:12:00.390724 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.390597    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3db823ab-4341-4ed0-ad70-412edeb3aefa-tls-cert\") pod \"authorino-54c4767776-4jqcd\" (UID: \"3db823ab-4341-4ed0-ad70-412edeb3aefa\") " pod="kuadrant-system/authorino-54c4767776-4jqcd"
Oct 30 00:12:00.392873 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.392855    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3db823ab-4341-4ed0-ad70-412edeb3aefa-tls-cert\") pod \"authorino-54c4767776-4jqcd\" (UID: \"3db823ab-4341-4ed0-ad70-412edeb3aefa\") " pod="kuadrant-system/authorino-54c4767776-4jqcd"
Oct 30 00:12:00.412772 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.412752    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvbcw\" (UniqueName: \"kubernetes.io/projected/3db823ab-4341-4ed0-ad70-412edeb3aefa-kube-api-access-jvbcw\") pod \"authorino-54c4767776-4jqcd\" (UID: \"3db823ab-4341-4ed0-ad70-412edeb3aefa\") " pod="kuadrant-system/authorino-54c4767776-4jqcd"
Oct 30 00:12:00.571770 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.571736    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54c4767776-4jqcd"
Oct 30 00:12:00.864422 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:00.864390    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54c4767776-4jqcd"]
Oct 30 00:12:00.866556 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:12:00.866529    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3db823ab_4341_4ed0_ad70_412edeb3aefa.slice/crio-0977c4bda97b4469ad9e391e675d824ea6038b09304c91ee28ef1e6e0fa65dda WatchSource:0}: Error finding container 0977c4bda97b4469ad9e391e675d824ea6038b09304c91ee28ef1e6e0fa65dda: Status 404 returned error can't find the container with id 0977c4bda97b4469ad9e391e675d824ea6038b09304c91ee28ef1e6e0fa65dda
Oct 30 00:12:01.218482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.218443    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54c4767776-4jqcd" event={"ID":"3db823ab-4341-4ed0-ad70-412edeb3aefa","Type":"ContainerStarted","Data":"abdbd76d5de5773b6a60fb5dd9538acb4c2e0e5445a8c35daafe09e3e449a994"}
Oct 30 00:12:01.218482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.218495    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54c4767776-4jqcd" event={"ID":"3db823ab-4341-4ed0-ad70-412edeb3aefa","Type":"ContainerStarted","Data":"0977c4bda97b4469ad9e391e675d824ea6038b09304c91ee28ef1e6e0fa65dda"}
Oct 30 00:12:01.222930 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.221688    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n6l7z"
Oct 30 00:12:01.222930 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.221718    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-msdnc" podUID="094be578-168a-44eb-b8bb-6dfceba40c5c" containerName="authorino" containerID="cri-o://6e87ce84a96cb099d9f677b98daa621a6b47dad8175b3e2955add0321fc8c99c" gracePeriod=30
Oct 30 00:12:01.222930 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.221891    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-msdnc" event={"ID":"094be578-168a-44eb-b8bb-6dfceba40c5c","Type":"ContainerStarted","Data":"6e87ce84a96cb099d9f677b98daa621a6b47dad8175b3e2955add0321fc8c99c"}
Oct 30 00:12:01.222930 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.221929    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-msdnc" event={"ID":"094be578-168a-44eb-b8bb-6dfceba40c5c","Type":"ContainerStarted","Data":"f5876ccd695188cfd0f5bcede4c6344d8c5a4e77f85e5848076e5230e171dbbc"}
Oct 30 00:12:01.229474 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.229452    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n6l7z"
Oct 30 00:12:01.257289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.257225    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-54c4767776-4jqcd" podStartSLOduration=0.976709626 podStartE2EDuration="1.257204865s" totalImagesPullingTime="280.495239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:12:01.256274253 +0000 UTC m=+430.781568129" watchObservedRunningTime="2026-10-30 00:12:01.257204865 +0000 UTC m=+430.782498744"
Oct 30 00:12:01.282578 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.282539    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-t8lbz"]
Oct 30 00:12:01.282766 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.282739    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-t8lbz" podUID="89f3f81c-652b-47c3-9b2b-71c389d43e10" containerName="authorino" containerID="cri-o://39d558261b43fae075ec8fe1901ea0998508e14b539ee592bd9486df04dc6a39" gracePeriod=30
Oct 30 00:12:01.297961 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.297907    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-msdnc" podStartSLOduration=1.760611327 podStartE2EDuration="2.297892856s" totalImagesPullingTime="537.281529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:12:01.295673213 +0000 UTC m=+430.820967091" watchObservedRunningTime="2026-10-30 00:12:01.297892856 +0000 UTC m=+430.823186734"
Oct 30 00:12:01.298650 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.298514    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-tls-cert\" (UniqueName: \"kubernetes.io/secret/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-tls-cert\") pod \"91e13842-ea9d-48f9-bec1-e6ae0ad4351a\" (UID: \"91e13842-ea9d-48f9-bec1-e6ae0ad4351a\") "
Oct 30 00:12:01.298650 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.298561    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-kube-api-access-lqz6n\" (UniqueName: \"kubernetes.io/projected/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-kube-api-access-lqz6n\") pod \"91e13842-ea9d-48f9-bec1-e6ae0ad4351a\" (UID: \"91e13842-ea9d-48f9-bec1-e6ae0ad4351a\") "
Oct 30 00:12:01.301715 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.301691    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-tls-cert" pod "91e13842-ea9d-48f9-bec1-e6ae0ad4351a" (UID: "91e13842-ea9d-48f9-bec1-e6ae0ad4351a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:12:01.302607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.302584    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-kube-api-access-lqz6n" pod "91e13842-ea9d-48f9-bec1-e6ae0ad4351a" (UID: "91e13842-ea9d-48f9-bec1-e6ae0ad4351a"). InnerVolumeSpecName "kube-api-access-lqz6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:12:01.399211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.399144    4441 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-tls-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:12:01.399211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.399171    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqz6n\" (UniqueName: \"kubernetes.io/projected/91e13842-ea9d-48f9-bec1-e6ae0ad4351a-kube-api-access-lqz6n\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:12:01.560224 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.560197    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-msdnc"
Oct 30 00:12:01.567762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.567742    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-t8lbz"
Oct 30 00:12:01.600689 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.600660    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f3f81c-652b-47c3-9b2b-71c389d43e10-kube-api-access-5jr45\" (UniqueName: \"kubernetes.io/projected/89f3f81c-652b-47c3-9b2b-71c389d43e10-kube-api-access-5jr45\") pod \"89f3f81c-652b-47c3-9b2b-71c389d43e10\" (UID: \"89f3f81c-652b-47c3-9b2b-71c389d43e10\") "
Oct 30 00:12:01.600887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.600735    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094be578-168a-44eb-b8bb-6dfceba40c5c-kube-api-access-jzlcv\" (UniqueName: \"kubernetes.io/projected/094be578-168a-44eb-b8bb-6dfceba40c5c-kube-api-access-jzlcv\") pod \"094be578-168a-44eb-b8bb-6dfceba40c5c\" (UID: \"094be578-168a-44eb-b8bb-6dfceba40c5c\") "
Oct 30 00:12:01.604069 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.604004    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f3f81c-652b-47c3-9b2b-71c389d43e10-kube-api-access-5jr45" pod "89f3f81c-652b-47c3-9b2b-71c389d43e10" (UID: "89f3f81c-652b-47c3-9b2b-71c389d43e10"). InnerVolumeSpecName "kube-api-access-5jr45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:12:01.605700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.605671    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094be578-168a-44eb-b8bb-6dfceba40c5c-kube-api-access-jzlcv" pod "094be578-168a-44eb-b8bb-6dfceba40c5c" (UID: "094be578-168a-44eb-b8bb-6dfceba40c5c"). InnerVolumeSpecName "kube-api-access-jzlcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:12:01.701601 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.701524    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jr45\" (UniqueName: \"kubernetes.io/projected/89f3f81c-652b-47c3-9b2b-71c389d43e10-kube-api-access-5jr45\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:12:01.701601 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:01.701565    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzlcv\" (UniqueName: \"kubernetes.io/projected/094be578-168a-44eb-b8bb-6dfceba40c5c-kube-api-access-jzlcv\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:12:02.227723 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.227691    4441 generic.go:356] "Generic (PLEG): container finished" podID="89f3f81c-652b-47c3-9b2b-71c389d43e10" containerID="39d558261b43fae075ec8fe1901ea0998508e14b539ee592bd9486df04dc6a39" exitCode=0
Oct 30 00:12:02.228196 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.227736    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-t8lbz"
Oct 30 00:12:02.228196 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.227769    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-t8lbz" event={"ID":"89f3f81c-652b-47c3-9b2b-71c389d43e10","Type":"ContainerDied","Data":"39d558261b43fae075ec8fe1901ea0998508e14b539ee592bd9486df04dc6a39"}
Oct 30 00:12:02.228196 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.227820    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-t8lbz" event={"ID":"89f3f81c-652b-47c3-9b2b-71c389d43e10","Type":"ContainerDied","Data":"0f0c54649ea83f46f8422d5e4ce329a21f200fca6bec3fd64405832d3754d98b"}
Oct 30 00:12:02.228196 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.227844    4441 scope.go:122] "RemoveContainer" containerID="39d558261b43fae075ec8fe1901ea0998508e14b539ee592bd9486df04dc6a39"
Oct 30 00:12:02.229299 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.229279    4441 generic.go:356] "Generic (PLEG): container finished" podID="094be578-168a-44eb-b8bb-6dfceba40c5c" containerID="6e87ce84a96cb099d9f677b98daa621a6b47dad8175b3e2955add0321fc8c99c" exitCode=0
Oct 30 00:12:02.229383 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.229328    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-msdnc"
Oct 30 00:12:02.229445 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.229381    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-msdnc" event={"ID":"094be578-168a-44eb-b8bb-6dfceba40c5c","Type":"ContainerDied","Data":"6e87ce84a96cb099d9f677b98daa621a6b47dad8175b3e2955add0321fc8c99c"}
Oct 30 00:12:02.229445 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.229421    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-msdnc" event={"ID":"094be578-168a-44eb-b8bb-6dfceba40c5c","Type":"ContainerDied","Data":"f5876ccd695188cfd0f5bcede4c6344d8c5a4e77f85e5848076e5230e171dbbc"}
Oct 30 00:12:02.229575 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.229451    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n6l7z"
Oct 30 00:12:02.238287 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.238267    4441 scope.go:122] "RemoveContainer" containerID="39d558261b43fae075ec8fe1901ea0998508e14b539ee592bd9486df04dc6a39"
Oct 30 00:12:02.238635 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:12:02.238609    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d558261b43fae075ec8fe1901ea0998508e14b539ee592bd9486df04dc6a39\": container with ID starting with 39d558261b43fae075ec8fe1901ea0998508e14b539ee592bd9486df04dc6a39 not found: ID does not exist" containerID="39d558261b43fae075ec8fe1901ea0998508e14b539ee592bd9486df04dc6a39"
Oct 30 00:12:02.238786 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.238645    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d558261b43fae075ec8fe1901ea0998508e14b539ee592bd9486df04dc6a39"} err="failed to get container status \"39d558261b43fae075ec8fe1901ea0998508e14b539ee592bd9486df04dc6a39\": rpc error: code = NotFound desc = could not find container \"39d558261b43fae075ec8fe1901ea0998508e14b539ee592bd9486df04dc6a39\": container with ID starting with 39d558261b43fae075ec8fe1901ea0998508e14b539ee592bd9486df04dc6a39 not found: ID does not exist"
Oct 30 00:12:02.238786 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.238670    4441 scope.go:122] "RemoveContainer" containerID="6e87ce84a96cb099d9f677b98daa621a6b47dad8175b3e2955add0321fc8c99c"
Oct 30 00:12:02.246796 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.246599    4441 scope.go:122] "RemoveContainer" containerID="6e87ce84a96cb099d9f677b98daa621a6b47dad8175b3e2955add0321fc8c99c"
Oct 30 00:12:02.246975 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:12:02.246950    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e87ce84a96cb099d9f677b98daa621a6b47dad8175b3e2955add0321fc8c99c\": container with ID starting with 6e87ce84a96cb099d9f677b98daa621a6b47dad8175b3e2955add0321fc8c99c not found: ID does not exist" containerID="6e87ce84a96cb099d9f677b98daa621a6b47dad8175b3e2955add0321fc8c99c"
Oct 30 00:12:02.247100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.246985    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e87ce84a96cb099d9f677b98daa621a6b47dad8175b3e2955add0321fc8c99c"} err="failed to get container status \"6e87ce84a96cb099d9f677b98daa621a6b47dad8175b3e2955add0321fc8c99c\": rpc error: code = NotFound desc = could not find container \"6e87ce84a96cb099d9f677b98daa621a6b47dad8175b3e2955add0321fc8c99c\": container with ID starting with 6e87ce84a96cb099d9f677b98daa621a6b47dad8175b3e2955add0321fc8c99c not found: ID does not exist"
Oct 30 00:12:02.302689 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.302611    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n6l7z"]
Oct 30 00:12:02.309075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.309045    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n6l7z"]
Oct 30 00:12:02.342064 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.340765    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-t8lbz"]
Oct 30 00:12:02.343408 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.343377    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-t8lbz"]
Oct 30 00:12:02.377386 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.377216    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-msdnc"]
Oct 30 00:12:02.378790 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.378768    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-msdnc"]
Oct 30 00:12:02.608214 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.608135    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094be578-168a-44eb-b8bb-6dfceba40c5c" path="/var/lib/kubelet/pods/094be578-168a-44eb-b8bb-6dfceba40c5c/volumes"
Oct 30 00:12:02.608672 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.608648    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f3f81c-652b-47c3-9b2b-71c389d43e10" path="/var/lib/kubelet/pods/89f3f81c-652b-47c3-9b2b-71c389d43e10/volumes"
Oct 30 00:12:02.609163 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:12:02.609144    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e13842-ea9d-48f9-bec1-e6ae0ad4351a" path="/var/lib/kubelet/pods/91e13842-ea9d-48f9-bec1-e6ae0ad4351a/volumes"
Oct 30 00:12:03.896992 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:12:03.896958    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:12:33.897337 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:12:33.897291    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:13:03.898270 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:13:03.898230    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:13:13.460658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:13.460619    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:13:17.631015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:17.630974    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:13:20.620068 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:20.619595    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:13:21.054551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.054513    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"]
Oct 30 00:13:21.055000 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.054975    4441 cpu_manager.go:414] "containerMap: removing container" podUID="094be578-168a-44eb-b8bb-6dfceba40c5c" containerName="authorino"
Oct 30 00:13:21.055000 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.055003    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094be578-168a-44eb-b8bb-6dfceba40c5c" containerName="authorino"
Oct 30 00:13:21.055272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.055063    4441 cpu_manager.go:414] "containerMap: removing container" podUID="89f3f81c-652b-47c3-9b2b-71c389d43e10" containerName="authorino"
Oct 30 00:13:21.055272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.055072    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f3f81c-652b-47c3-9b2b-71c389d43e10" containerName="authorino"
Oct 30 00:13:21.055272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.055160    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f3f81c-652b-47c3-9b2b-71c389d43e10" containerName="authorino"
Oct 30 00:13:21.055272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.055173    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="094be578-168a-44eb-b8bb-6dfceba40c5c" containerName="authorino"
Oct 30 00:13:21.057541 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.057513    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.076429 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.074536    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 00:13:21.086070 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.085830    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 00:13:21.086070 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.085902    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vrftc\""
Oct 30 00:13:21.086070 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.085904    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-external-token-limit-simulated-kserve-self-signed-certs\""
Oct 30 00:13:21.088327 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.088302    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"]
Oct 30 00:13:21.194458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.194416    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a70f3491-6506-456f-ab45-90b1291f086d-tmp-dir\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.194829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.194808    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a70f3491-6506-456f-ab45-90b1291f086d-tls-certs\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.195063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.195013    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a70f3491-6506-456f-ab45-90b1291f086d-home\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.195254 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.195235    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a70f3491-6506-456f-ab45-90b1291f086d-model-cache\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.195417 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.195401    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a70f3491-6506-456f-ab45-90b1291f086d-dshm\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.195649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.195630    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdkpw\" (UniqueName: \"kubernetes.io/projected/a70f3491-6506-456f-ab45-90b1291f086d-kube-api-access-mdkpw\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.296505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.296474    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a70f3491-6506-456f-ab45-90b1291f086d-tls-certs\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.296505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.296510    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a70f3491-6506-456f-ab45-90b1291f086d-home\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.296765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.296529    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a70f3491-6506-456f-ab45-90b1291f086d-model-cache\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.296765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.296553    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a70f3491-6506-456f-ab45-90b1291f086d-dshm\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.296765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.296613    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdkpw\" (UniqueName: \"kubernetes.io/projected/a70f3491-6506-456f-ab45-90b1291f086d-kube-api-access-mdkpw\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.296765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.296695    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a70f3491-6506-456f-ab45-90b1291f086d-tmp-dir\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.297041 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.297004    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a70f3491-6506-456f-ab45-90b1291f086d-home\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.297120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.297061    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a70f3491-6506-456f-ab45-90b1291f086d-model-cache\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.297176 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.297158    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a70f3491-6506-456f-ab45-90b1291f086d-tmp-dir\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.298780 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.298757    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a70f3491-6506-456f-ab45-90b1291f086d-dshm\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.299127 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.299108    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a70f3491-6506-456f-ab45-90b1291f086d-tls-certs\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.319070 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.318995    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdkpw\" (UniqueName: \"kubernetes.io/projected/a70f3491-6506-456f-ab45-90b1291f086d-kube-api-access-mdkpw\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l\" (UID: \"a70f3491-6506-456f-ab45-90b1291f086d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.374671 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.374645    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:21.667755 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:21.667726    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"]
Oct 30 00:13:21.670466 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:13:21.670437    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70f3491_6506_456f_ab45_90b1291f086d.slice/crio-2d5249278e2ef313740124fed18ffab7283c7c96866e22656e275f84361035a7 WatchSource:0}: Error finding container 2d5249278e2ef313740124fed18ffab7283c7c96866e22656e275f84361035a7: Status 404 returned error can't find the container with id 2d5249278e2ef313740124fed18ffab7283c7c96866e22656e275f84361035a7
Oct 30 00:13:22.558674 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:22.558632    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l" event={"ID":"a70f3491-6506-456f-ab45-90b1291f086d","Type":"ContainerStarted","Data":"2d5249278e2ef313740124fed18ffab7283c7c96866e22656e275f84361035a7"}
Oct 30 00:13:23.565673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:23.565636    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l" event={"ID":"a70f3491-6506-456f-ab45-90b1291f086d","Type":"ContainerStarted","Data":"ddfca6236a6d5ec12fdef73c6293163eb0b481b744fe0226498c98155fe89ddd"}
Oct 30 00:13:23.609119 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:23.609071    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l" podStartSLOduration=1.524666665 podStartE2EDuration="2.609054938s" totalImagesPullingTime="1.084388273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:23.607445426 +0000 UTC m=+513.132739304" watchObservedRunningTime="2026-10-30 00:13:23.609054938 +0000 UTC m=+513.134348835"
Oct 30 00:13:24.375719 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:24.375682    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:25.781429 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.781393    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"]
Oct 30 00:13:25.783895 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.783875    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.809673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.809648    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 00:13:25.811908 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.811883    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"]
Oct 30 00:13:25.832438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.832415    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.832545 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.832455    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.832545 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.832486    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.832615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.832596    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjstg\" (UniqueName: \"kubernetes.io/projected/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-kube-api-access-wjstg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.832649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.832634    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.832684 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.832657    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.933613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.933591    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjstg\" (UniqueName: \"kubernetes.io/projected/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-kube-api-access-wjstg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.933727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.933624    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.933727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.933644    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.933727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.933677    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.933727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.933705    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.933727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.933726    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.934130 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.934105    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.934211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.934127    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.934211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.934141    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.936040 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.936012    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.936220 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.936203    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:25.955708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:25.955685    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjstg\" (UniqueName: \"kubernetes.io/projected/a40101d4-2789-4fa1-b9cf-6fd2e518efe5-kube-api-access-wjstg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r\" (UID: \"a40101d4-2789-4fa1-b9cf-6fd2e518efe5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:26.095587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:26.095518    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:26.397307 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:26.394264    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"]
Oct 30 00:13:26.400779 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:13:26.400752    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40101d4_2789_4fa1_b9cf_6fd2e518efe5.slice/crio-702e6aeb240104b2cb4edbe681349833ed10740e6c6895547c4ddfda14d0acdc WatchSource:0}: Error finding container 702e6aeb240104b2cb4edbe681349833ed10740e6c6895547c4ddfda14d0acdc: Status 404 returned error can't find the container with id 702e6aeb240104b2cb4edbe681349833ed10740e6c6895547c4ddfda14d0acdc
Oct 30 00:13:26.579696 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:26.579664    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r" event={"ID":"a40101d4-2789-4fa1-b9cf-6fd2e518efe5","Type":"ContainerStarted","Data":"702e6aeb240104b2cb4edbe681349833ed10740e6c6895547c4ddfda14d0acdc"}
Oct 30 00:13:27.007838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:27.007795    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:13:27.595470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:27.595420    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r" event={"ID":"a40101d4-2789-4fa1-b9cf-6fd2e518efe5","Type":"ContainerStarted","Data":"160960b200b989e3bbf6628bde1bab4fec5fa7fa346b499ca48752dad96acf63"}
Oct 30 00:13:27.636655 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:27.636597    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r" podStartSLOduration=2.440863185 podStartE2EDuration="2.636578868s" totalImagesPullingTime="195.715683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:27.634719591 +0000 UTC m=+517.160013468" watchObservedRunningTime="2026-10-30 00:13:27.636578868 +0000 UTC m=+517.161872746"
Oct 30 00:13:28.096480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:28.096450    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:29.393332 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.393297    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"]
Oct 30 00:13:29.398750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.398724    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.416467 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.416445    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"]
Oct 30 00:13:29.425366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.425333    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 00:13:29.465487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.465454    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7100f212-f2df-48aa-9895-d51333bc10ad-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.465650 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.465537    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7100f212-f2df-48aa-9895-d51333bc10ad-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.465650 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.465567    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v4fq\" (UniqueName: \"kubernetes.io/projected/7100f212-f2df-48aa-9895-d51333bc10ad-kube-api-access-4v4fq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.465743 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.465647    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7100f212-f2df-48aa-9895-d51333bc10ad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.465743 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.465689    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7100f212-f2df-48aa-9895-d51333bc10ad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.465835 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.465762    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7100f212-f2df-48aa-9895-d51333bc10ad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.566240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.566199    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7100f212-f2df-48aa-9895-d51333bc10ad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.566240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.566246    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7100f212-f2df-48aa-9895-d51333bc10ad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.566487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.566282    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7100f212-f2df-48aa-9895-d51333bc10ad-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.566487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.566331    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7100f212-f2df-48aa-9895-d51333bc10ad-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.566487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.566347    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v4fq\" (UniqueName: \"kubernetes.io/projected/7100f212-f2df-48aa-9895-d51333bc10ad-kube-api-access-4v4fq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.566487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.566378    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7100f212-f2df-48aa-9895-d51333bc10ad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.566704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.566678    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7100f212-f2df-48aa-9895-d51333bc10ad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.566763 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.566685    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7100f212-f2df-48aa-9895-d51333bc10ad-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.566763 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.566732    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7100f212-f2df-48aa-9895-d51333bc10ad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.568727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.568703    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7100f212-f2df-48aa-9895-d51333bc10ad-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.569048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.568998    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7100f212-f2df-48aa-9895-d51333bc10ad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.589098 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.589077    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v4fq\" (UniqueName: \"kubernetes.io/projected/7100f212-f2df-48aa-9895-d51333bc10ad-kube-api-access-4v4fq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hx728\" (UID: \"7100f212-f2df-48aa-9895-d51333bc10ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:29.716272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:29.716184    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:30.025007 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:30.024972    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"]
Oct 30 00:13:30.028616 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:13:30.028579    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7100f212_f2df_48aa_9895_d51333bc10ad.slice/crio-aaaf23cc9c2532c27429453fbb949dee8d7fabfa921c976dd0cba09d2dfe2853 WatchSource:0}: Error finding container aaaf23cc9c2532c27429453fbb949dee8d7fabfa921c976dd0cba09d2dfe2853: Status 404 returned error can't find the container with id aaaf23cc9c2532c27429453fbb949dee8d7fabfa921c976dd0cba09d2dfe2853
Oct 30 00:13:30.610998 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:30.610933    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728" event={"ID":"7100f212-f2df-48aa-9895-d51333bc10ad","Type":"ContainerStarted","Data":"f74f483246cc3877631fd10d2d30ae233b852d5bb4ea0865400bf02109cebf93"}
Oct 30 00:13:30.610998 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:30.610978    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728" event={"ID":"7100f212-f2df-48aa-9895-d51333bc10ad","Type":"ContainerStarted","Data":"aaaf23cc9c2532c27429453fbb949dee8d7fabfa921c976dd0cba09d2dfe2853"}
Oct 30 00:13:30.651598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:30.651551    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728" podStartSLOduration=1.45192538 podStartE2EDuration="1.651535046s" totalImagesPullingTime="199.609666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:30.650866008 +0000 UTC m=+520.176159886" watchObservedRunningTime="2026-10-30 00:13:30.651535046 +0000 UTC m=+520.176828924"
Oct 30 00:13:30.716564 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:30.716539    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:31.375138 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:31.375103    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:31.387412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:31.387391    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:31.625956 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:31.625875    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l"
Oct 30 00:13:32.212751 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:32.212711    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:13:33.898960 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:13:33.898918    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:13:34.899037 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.898988    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"]
Oct 30 00:13:34.901713 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.901692    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:34.917251 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.917227    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:34.917354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.917269    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:34.917354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.917301    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2frxn\" (UniqueName: \"kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-kube-api-access-2frxn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:34.917450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.917411    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:34.917495 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.917449    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:34.917533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.917507    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:34.917570 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.917530    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98b0e606-d290-40ee-a057-77633a3e07bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:34.917603 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.917583    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:34.917633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.917611    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98b0e606-d290-40ee-a057-77633a3e07bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:34.918162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.918147    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 00:13:34.918231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.918185    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tgqt4\""
Oct 30 00:13:34.931313 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:34.931290    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"]
Oct 30 00:13:35.018760 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.018735    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.018861 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.018770    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.018861 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.018792    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2frxn\" (UniqueName: \"kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-kube-api-access-2frxn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.018861 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.018817    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.018861 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.018850    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.019100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.018875    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.019100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.018898    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98b0e606-d290-40ee-a057-77633a3e07bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.019100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.018927    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.019100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.018974    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98b0e606-d290-40ee-a057-77633a3e07bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.019310 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.019272    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.019369 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.019354    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.019413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.019393    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.019475 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.019452    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.019665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.019648    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98b0e606-d290-40ee-a057-77633a3e07bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.021191 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.021170    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.021416 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.021398    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98b0e606-d290-40ee-a057-77633a3e07bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.040696 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.040679    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.040934 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.040917    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2frxn\" (UniqueName: \"kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-kube-api-access-2frxn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n8j6t\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.214583 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.214506    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:35.522851 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.522728    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"]
Oct 30 00:13:35.525039 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:13:35.525004    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b0e606_d290_40ee_a057_77633a3e07bf.slice/crio-8343d3d3167ea9bf29f1f44bbcb4fac62d6aa8e26ce502011f184c81f37def5a WatchSource:0}: Error finding container 8343d3d3167ea9bf29f1f44bbcb4fac62d6aa8e26ce502011f184c81f37def5a: Status 404 returned error can't find the container with id 8343d3d3167ea9bf29f1f44bbcb4fac62d6aa8e26ce502011f184c81f37def5a
Oct 30 00:13:35.635068 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:35.635039    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t" event={"ID":"98b0e606-d290-40ee-a057-77633a3e07bf","Type":"ContainerStarted","Data":"8343d3d3167ea9bf29f1f44bbcb4fac62d6aa8e26ce502011f184c81f37def5a"}
Oct 30 00:13:36.095847 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:36.095815    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:36.108424 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:36.108404    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:36.656679 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:36.656646    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r"
Oct 30 00:13:38.857712 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:38.857683    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:13:38.857982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:38.857751    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:13:38.857982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:38.857781    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:13:39.657122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:39.657086    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t" event={"ID":"98b0e606-d290-40ee-a057-77633a3e07bf","Type":"ContainerStarted","Data":"f1c7d75d477924d88acd073eaa280b0d43d5402ccff2db13891e26754317a6e8"}
Oct 30 00:13:39.713472 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:39.713425    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t" podStartSLOduration=2.38335206 podStartE2EDuration="5.713411847s" totalImagesPullingTime="3.330059787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:39.712504023 +0000 UTC m=+529.237797902" watchObservedRunningTime="2026-10-30 00:13:39.713411847 +0000 UTC m=+529.238705723"
Oct 30 00:13:39.717228 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:39.717200    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:39.747328 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:39.747291    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:40.215486 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:40.215453    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:40.216642 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:40.216619    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 00:13:40.216693 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:40.216668    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t" podUID="98b0e606-d290-40ee-a057-77633a3e07bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 00:13:40.673160 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:40.673130    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728"
Oct 30 00:13:41.215650 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:41.215612    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 00:13:41.216052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:41.215666    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t" podUID="98b0e606-d290-40ee-a057-77633a3e07bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 00:13:42.215339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:42.215307    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 00:13:42.215516 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:42.215351    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t" podUID="98b0e606-d290-40ee-a057-77633a3e07bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 00:13:43.215697 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:43.215670    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 00:13:43.216089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:43.215716    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t" podUID="98b0e606-d290-40ee-a057-77633a3e07bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 00:13:44.214926 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:44.214899    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 00:13:44.215076 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:44.214949    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t" podUID="98b0e606-d290-40ee-a057-77633a3e07bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 00:13:45.214838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:45.214794    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 00:13:45.215376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:45.214847    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t" podUID="98b0e606-d290-40ee-a057-77633a3e07bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 00:13:46.215768 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:46.215727    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 00:13:46.216174 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:46.215794    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t" podUID="98b0e606-d290-40ee-a057-77633a3e07bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 00:13:47.297328 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.297294    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-748768979-s8jhn"]
Oct 30 00:13:47.299870 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.299849    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-748768979-s8jhn"
Oct 30 00:13:47.317366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.316810    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-748768979-s8jhn"]
Oct 30 00:13:47.333303 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.333273    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ptjd\" (UniqueName: \"kubernetes.io/projected/fa78720d-333c-4f7a-891f-1d6e4f120472-kube-api-access-6ptjd\") pod \"authorino-748768979-s8jhn\" (UID: \"fa78720d-333c-4f7a-891f-1d6e4f120472\") " pod="kuadrant-system/authorino-748768979-s8jhn"
Oct 30 00:13:47.333419 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.333319    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa78720d-333c-4f7a-891f-1d6e4f120472-tls-cert\") pod \"authorino-748768979-s8jhn\" (UID: \"fa78720d-333c-4f7a-891f-1d6e4f120472\") " pod="kuadrant-system/authorino-748768979-s8jhn"
Oct 30 00:13:47.386243 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.386203    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:47.386988 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.386965    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:47.388167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.388145    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:13:47.434435 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.434406    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ptjd\" (UniqueName: \"kubernetes.io/projected/fa78720d-333c-4f7a-891f-1d6e4f120472-kube-api-access-6ptjd\") pod \"authorino-748768979-s8jhn\" (UID: \"fa78720d-333c-4f7a-891f-1d6e4f120472\") " pod="kuadrant-system/authorino-748768979-s8jhn"
Oct 30 00:13:47.434552 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.434453    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa78720d-333c-4f7a-891f-1d6e4f120472-tls-cert\") pod \"authorino-748768979-s8jhn\" (UID: \"fa78720d-333c-4f7a-891f-1d6e4f120472\") " pod="kuadrant-system/authorino-748768979-s8jhn"
Oct 30 00:13:47.436891 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.436866    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa78720d-333c-4f7a-891f-1d6e4f120472-tls-cert\") pod \"authorino-748768979-s8jhn\" (UID: \"fa78720d-333c-4f7a-891f-1d6e4f120472\") " pod="kuadrant-system/authorino-748768979-s8jhn"
Oct 30 00:13:47.456560 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.456516    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ptjd\" (UniqueName: \"kubernetes.io/projected/fa78720d-333c-4f7a-891f-1d6e4f120472-kube-api-access-6ptjd\") pod \"authorino-748768979-s8jhn\" (UID: \"fa78720d-333c-4f7a-891f-1d6e4f120472\") " pod="kuadrant-system/authorino-748768979-s8jhn"
Oct 30 00:13:47.614914 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.614836    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-748768979-s8jhn"
Oct 30 00:13:47.913090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:47.912786    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-748768979-s8jhn"]
Oct 30 00:13:47.915104 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:13:47.915072    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa78720d_333c_4f7a_891f_1d6e4f120472.slice/crio-0ebf68e38a700f4342e702d737ba16275785ae1e5e2d9147365a7f4324ba8848 WatchSource:0}: Error finding container 0ebf68e38a700f4342e702d737ba16275785ae1e5e2d9147365a7f4324ba8848: Status 404 returned error can't find the container with id 0ebf68e38a700f4342e702d737ba16275785ae1e5e2d9147365a7f4324ba8848
Oct 30 00:13:48.710186 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:48.710153    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-748768979-s8jhn" event={"ID":"fa78720d-333c-4f7a-891f-1d6e4f120472","Type":"ContainerStarted","Data":"0ebf68e38a700f4342e702d737ba16275785ae1e5e2d9147365a7f4324ba8848"}
Oct 30 00:13:49.716433 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:49.716397    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-748768979-s8jhn" event={"ID":"fa78720d-333c-4f7a-891f-1d6e4f120472","Type":"ContainerStarted","Data":"f41845dd968d19af211fd16f7169d244ded2e491ab1f682aaa1963fb9b815a3d"}
Oct 30 00:13:49.757934 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:49.757878    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-748768979-s8jhn" podStartSLOduration=1.9490360199999999 podStartE2EDuration="2.757863516s" totalImagesPullingTime="808.827496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:49.756431077 +0000 UTC m=+539.281724955" watchObservedRunningTime="2026-10-30 00:13:49.757863516 +0000 UTC m=+539.283157393"
Oct 30 00:13:49.790138 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:49.790097    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54c4767776-4jqcd"]
Oct 30 00:13:49.790345 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:49.790315    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-54c4767776-4jqcd" podUID="3db823ab-4341-4ed0-ad70-412edeb3aefa" containerName="authorino" containerID="cri-o://abdbd76d5de5773b6a60fb5dd9538acb4c2e0e5445a8c35daafe09e3e449a994" gracePeriod=30
Oct 30 00:13:49.946236 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:49.946180    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"]
Oct 30 00:13:49.950453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:49.950423    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:49.985075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:49.985039    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"]
Oct 30 00:13:49.985507 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:49.985481    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"]
Oct 30 00:13:49.995898 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:49.995869    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.004486 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.004461    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"]
Oct 30 00:13:50.057117 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057078    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.057302 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057142    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.057302 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057218    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.057423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057311    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjcc\" (UniqueName: \"kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-kube-api-access-8vjcc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.057423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057337    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972bef50-60cd-4b01-85ab-c17359d1327b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.057532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057444    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.057532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057475    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.057532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057501    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972bef50-60cd-4b01-85ab-c17359d1327b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.057685 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057567    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnpcz\" (UniqueName: \"kubernetes.io/projected/972bef50-60cd-4b01-85ab-c17359d1327b-kube-api-access-jnpcz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.057685 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057646    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.057685 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057674    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.057829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057751    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27789fe-70fb-4912-8873-aabb7694db35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.057829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057779    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.057829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057810    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972bef50-60cd-4b01-85ab-c17359d1327b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.057982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057886    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.057982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057911    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.057982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057931    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.057982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.057957    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27789fe-70fb-4912-8873-aabb7694db35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.077894 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.077874    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54c4767776-4jqcd"
Oct 30 00:13:50.159341 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159309    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3db823ab-4341-4ed0-ad70-412edeb3aefa-tls-cert\" (UniqueName: \"kubernetes.io/secret/3db823ab-4341-4ed0-ad70-412edeb3aefa-tls-cert\") pod \"3db823ab-4341-4ed0-ad70-412edeb3aefa\" (UID: \"3db823ab-4341-4ed0-ad70-412edeb3aefa\") "
Oct 30 00:13:50.159557 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159393    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db823ab-4341-4ed0-ad70-412edeb3aefa-kube-api-access-jvbcw\" (UniqueName: \"kubernetes.io/projected/3db823ab-4341-4ed0-ad70-412edeb3aefa-kube-api-access-jvbcw\") pod \"3db823ab-4341-4ed0-ad70-412edeb3aefa\" (UID: \"3db823ab-4341-4ed0-ad70-412edeb3aefa\") "
Oct 30 00:13:50.159651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159591    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnpcz\" (UniqueName: \"kubernetes.io/projected/972bef50-60cd-4b01-85ab-c17359d1327b-kube-api-access-jnpcz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.159651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159633    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.159764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159658    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.159764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159698    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27789fe-70fb-4912-8873-aabb7694db35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.159764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159726    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.160039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159766    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972bef50-60cd-4b01-85ab-c17359d1327b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.160039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159803    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.160039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159825    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.160039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159856    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.160039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159889    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27789fe-70fb-4912-8873-aabb7694db35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.160039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159935    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.160039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.159981    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.160039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.160004    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.160438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.160072    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjcc\" (UniqueName: \"kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-kube-api-access-8vjcc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.160438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.160102    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972bef50-60cd-4b01-85ab-c17359d1327b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.160438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.160191    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.160438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.160219    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.160438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.160243    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972bef50-60cd-4b01-85ab-c17359d1327b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.160693 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.160442    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.163442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.161827    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.163442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.162524    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972bef50-60cd-4b01-85ab-c17359d1327b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.163442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.162546    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db823ab-4341-4ed0-ad70-412edeb3aefa-kube-api-access-jvbcw" pod "3db823ab-4341-4ed0-ad70-412edeb3aefa" (UID: "3db823ab-4341-4ed0-ad70-412edeb3aefa"). InnerVolumeSpecName "kube-api-access-jvbcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:13:50.163442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.162792    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.163442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.162921    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972bef50-60cd-4b01-85ab-c17359d1327b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.163442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.162928    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.163442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.163306    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.163881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.163479    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.163881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.163491    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27789fe-70fb-4912-8873-aabb7694db35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.163881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.163702    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27789fe-70fb-4912-8873-aabb7694db35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.163881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.163784    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.164393 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.164360    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.165575 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.165548    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.165659 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.165548    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972bef50-60cd-4b01-85ab-c17359d1327b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.173776 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.173719    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3db823ab-4341-4ed0-ad70-412edeb3aefa-tls-cert" pod "3db823ab-4341-4ed0-ad70-412edeb3aefa" (UID: "3db823ab-4341-4ed0-ad70-412edeb3aefa"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:13:50.182444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.182420    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972bef50-60cd-4b01-85ab-c17359d1327b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.182543 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.182526    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnpcz\" (UniqueName: \"kubernetes.io/projected/972bef50-60cd-4b01-85ab-c17359d1327b-kube-api-access-jnpcz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mjckv\" (UID: \"972bef50-60cd-4b01-85ab-c17359d1327b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.183547 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.183525    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjcc\" (UniqueName: \"kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-kube-api-access-8vjcc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.184445 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.184423    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wrv7z\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.261525 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.261490    4441 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3db823ab-4341-4ed0-ad70-412edeb3aefa-tls-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:13:50.261525 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.261520    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvbcw\" (UniqueName: \"kubernetes.io/projected/3db823ab-4341-4ed0-ad70-412edeb3aefa-kube-api-access-jvbcw\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:13:50.268407 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.268372    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:50.316283 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.316252    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:50.584941 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.584909    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"]
Oct 30 00:13:50.588133 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:13:50.588102    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972bef50_60cd_4b01_85ab_c17359d1327b.slice/crio-b4d9d596c50db477d9a973ec8a6908fda561eaaf349afd03bd58fccbaed15c55 WatchSource:0}: Error finding container b4d9d596c50db477d9a973ec8a6908fda561eaaf349afd03bd58fccbaed15c55: Status 404 returned error can't find the container with id b4d9d596c50db477d9a973ec8a6908fda561eaaf349afd03bd58fccbaed15c55
Oct 30 00:13:50.596113 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.596082    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:13:50.596222 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.596182    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:13:50.596285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.596228    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:13:50.638239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.638074    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"]
Oct 30 00:13:50.648203 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.648167    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:13:50.648419 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.648395    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:13:50.648562 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.648542    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:13:50.725594 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.725172    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z" event={"ID":"e27789fe-70fb-4912-8873-aabb7694db35","Type":"ContainerStarted","Data":"2b6385531efca2ee0938acdb560c190c55ded62bbc892cb404ddf7ca7900a2fe"}
Oct 30 00:13:50.728723 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.728595    4441 generic.go:356] "Generic (PLEG): container finished" podID="3db823ab-4341-4ed0-ad70-412edeb3aefa" containerID="abdbd76d5de5773b6a60fb5dd9538acb4c2e0e5445a8c35daafe09e3e449a994" exitCode=0
Oct 30 00:13:50.728723 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.728669    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54c4767776-4jqcd" event={"ID":"3db823ab-4341-4ed0-ad70-412edeb3aefa","Type":"ContainerDied","Data":"abdbd76d5de5773b6a60fb5dd9538acb4c2e0e5445a8c35daafe09e3e449a994"}
Oct 30 00:13:50.728723 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.728699    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54c4767776-4jqcd" event={"ID":"3db823ab-4341-4ed0-ad70-412edeb3aefa","Type":"ContainerDied","Data":"0977c4bda97b4469ad9e391e675d824ea6038b09304c91ee28ef1e6e0fa65dda"}
Oct 30 00:13:50.728723 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.728718    4441 scope.go:122] "RemoveContainer" containerID="abdbd76d5de5773b6a60fb5dd9538acb4c2e0e5445a8c35daafe09e3e449a994"
Oct 30 00:13:50.728932 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.728859    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54c4767776-4jqcd"
Oct 30 00:13:50.732010 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.731980    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv" event={"ID":"972bef50-60cd-4b01-85ab-c17359d1327b","Type":"ContainerStarted","Data":"a0516120a6764b1e7c13a92d5f01dc9f624b53fd59dd04d734d814560484cf82"}
Oct 30 00:13:50.732169 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.732060    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv" event={"ID":"972bef50-60cd-4b01-85ab-c17359d1327b","Type":"ContainerStarted","Data":"b4d9d596c50db477d9a973ec8a6908fda561eaaf349afd03bd58fccbaed15c55"}
Oct 30 00:13:50.741796 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.741762    4441 scope.go:122] "RemoveContainer" containerID="abdbd76d5de5773b6a60fb5dd9538acb4c2e0e5445a8c35daafe09e3e449a994"
Oct 30 00:13:50.742240 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:13:50.742214    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abdbd76d5de5773b6a60fb5dd9538acb4c2e0e5445a8c35daafe09e3e449a994\": container with ID starting with abdbd76d5de5773b6a60fb5dd9538acb4c2e0e5445a8c35daafe09e3e449a994 not found: ID does not exist" containerID="abdbd76d5de5773b6a60fb5dd9538acb4c2e0e5445a8c35daafe09e3e449a994"
Oct 30 00:13:50.742323 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.742250    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abdbd76d5de5773b6a60fb5dd9538acb4c2e0e5445a8c35daafe09e3e449a994"} err="failed to get container status \"abdbd76d5de5773b6a60fb5dd9538acb4c2e0e5445a8c35daafe09e3e449a994\": rpc error: code = NotFound desc = could not find container \"abdbd76d5de5773b6a60fb5dd9538acb4c2e0e5445a8c35daafe09e3e449a994\": container with ID starting with abdbd76d5de5773b6a60fb5dd9538acb4c2e0e5445a8c35daafe09e3e449a994 not found: ID does not exist"
Oct 30 00:13:50.790773 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.790744    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54c4767776-4jqcd"]
Oct 30 00:13:50.801861 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.801822    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-54c4767776-4jqcd"]
Oct 30 00:13:50.870542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:50.870408    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv" podStartSLOduration=1.8703897999999999 podStartE2EDuration="1.8703898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:50.86797201 +0000 UTC m=+540.393265920" watchObservedRunningTime="2026-10-30 00:13:50.8703898 +0000 UTC m=+540.395683680"
Oct 30 00:13:51.271977 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:51.270844    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:51.271977 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:51.270981    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mjckv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 00:13:51.271977 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:51.271056    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv" podUID="972bef50-60cd-4b01-85ab-c17359d1327b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 00:13:51.781792 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:51.781728    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z" event={"ID":"e27789fe-70fb-4912-8873-aabb7694db35","Type":"ContainerStarted","Data":"af2c25ae150a2438ebb9227bc9c97fbe24f8f2354cef6adad834310caeeb1102"}
Oct 30 00:13:51.845049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:51.840685    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z" podStartSLOduration=2.8406626900000003 podStartE2EDuration="2.84066269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:51.838201435 +0000 UTC m=+541.363495315" watchObservedRunningTime="2026-10-30 00:13:51.84066269 +0000 UTC m=+541.365956568"
Oct 30 00:13:52.268976 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:52.268938    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mjckv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 00:13:52.269342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:52.269308    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv" podUID="972bef50-60cd-4b01-85ab-c17359d1327b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 00:13:52.317118 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:52.317083    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:52.318314 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:52.318292    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 00:13:52.318484 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:52.318465    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z" podUID="e27789fe-70fb-4912-8873-aabb7694db35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 00:13:52.607573 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:52.607495    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db823ab-4341-4ed0-ad70-412edeb3aefa" path="/var/lib/kubelet/pods/3db823ab-4341-4ed0-ad70-412edeb3aefa/volumes"
Oct 30 00:13:53.270049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:53.269592    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mjckv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 00:13:53.270049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:53.269666    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv" podUID="972bef50-60cd-4b01-85ab-c17359d1327b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 00:13:53.317854 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:53.317506    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 00:13:53.317854 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:53.317557    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z" podUID="e27789fe-70fb-4912-8873-aabb7694db35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 00:13:54.269451 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:54.269418    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mjckv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 00:13:54.269713 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:54.269687    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv" podUID="972bef50-60cd-4b01-85ab-c17359d1327b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 00:13:54.317476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:54.317442    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 00:13:54.318057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:54.318011    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z" podUID="e27789fe-70fb-4912-8873-aabb7694db35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 00:13:55.269941 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:55.269905    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mjckv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 00:13:55.270236 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:55.270210    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv" podUID="972bef50-60cd-4b01-85ab-c17359d1327b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 00:13:55.317503 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:55.317474    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 00:13:55.317944 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:55.317919    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z" podUID="e27789fe-70fb-4912-8873-aabb7694db35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 00:13:56.272431 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:56.272391    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mjckv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 00:13:56.272701 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:56.272672    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv" podUID="972bef50-60cd-4b01-85ab-c17359d1327b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 00:13:56.317509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:56.317469    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 00:13:56.317509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:56.317513    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z" podUID="e27789fe-70fb-4912-8873-aabb7694db35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 00:13:57.269785 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:57.269745    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mjckv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 00:13:57.269965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:57.269815    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv" podUID="972bef50-60cd-4b01-85ab-c17359d1327b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 00:13:57.317678 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:57.317637    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 00:13:57.318079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:57.317706    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z" podUID="e27789fe-70fb-4912-8873-aabb7694db35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 00:13:58.268970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:58.268935    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mjckv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 00:13:58.269176 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:58.268996    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv" podUID="972bef50-60cd-4b01-85ab-c17359d1327b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 00:13:58.316906 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:58.316875    4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 00:13:58.317005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:58.316925    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z" podUID="e27789fe-70fb-4912-8873-aabb7694db35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 00:13:59.427625 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:59.427594    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:59.428806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:59.428786    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:59.429391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:59.429372    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mjckv"
Oct 30 00:13:59.675316 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:59.675289    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:59.676175 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:59.676140    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:13:59.676669 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:13:59.676650    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:14:03.898444 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:14:03.898405    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:14:28.308539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.308505    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"]
Oct 30 00:14:28.310610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.308942    4441 cpu_manager.go:414] "containerMap: removing container" podUID="3db823ab-4341-4ed0-ad70-412edeb3aefa" containerName="authorino"
Oct 30 00:14:28.310610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.308957    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db823ab-4341-4ed0-ad70-412edeb3aefa" containerName="authorino"
Oct 30 00:14:28.310610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.309017    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db823ab-4341-4ed0-ad70-412edeb3aefa" containerName="authorino"
Oct 30 00:14:28.311508 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.311490    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.343589 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.343562    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6c45bb-openshift-default-dockercfg-d5ghz\""
Oct 30 00:14:28.345241 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.345221    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"]
Oct 30 00:14:28.357793 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.357769    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.357914 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.357797    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.357914 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.357822    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.357985 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.357930    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.357985 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.357964    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.357985 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.357982    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37484057-61d0-44a9-96db-4d814905649b-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.358136 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.358006    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq9m8\" (UniqueName: \"kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-kube-api-access-gq9m8\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.358136 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.358070    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.358205 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.358168    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37484057-61d0-44a9-96db-4d814905649b-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.460137 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.460080    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.461440 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.461400    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.461652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.461634    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.461813 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.461800    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37484057-61d0-44a9-96db-4d814905649b-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.462491 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.462474    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq9m8\" (UniqueName: \"kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-kube-api-access-gq9m8\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.462968 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.462948    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.463376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.463360    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37484057-61d0-44a9-96db-4d814905649b-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.464288 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.464242    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37484057-61d0-44a9-96db-4d814905649b-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.464656 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.464620    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.464868 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.464850    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.465034 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.464968    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.465173 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.465158    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.465598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.465582    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.466077 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.465910    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.468190 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.468159    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.468839 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.468811    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37484057-61d0-44a9-96db-4d814905649b-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.482697 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.482675    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.484860 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.484835    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq9m8\" (UniqueName: \"kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-kube-api-access-gq9m8\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.627497 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.627422    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:28.972602 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.972518    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"]
Oct 30 00:14:28.977160 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:14:28.977133    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37484057_61d0_44a9_96db_4d814905649b.slice/crio-1b7f480b242b0cac038ceb48593980223c2d3708816d0e1757ba0df93c1ebbf6 WatchSource:0}: Error finding container 1b7f480b242b0cac038ceb48593980223c2d3708816d0e1757ba0df93c1ebbf6: Status 404 returned error can't find the container with id 1b7f480b242b0cac038ceb48593980223c2d3708816d0e1757ba0df93c1ebbf6
Oct 30 00:14:28.979743 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.979709    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:28.979850 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.979775    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:28.979850 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.979812    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:28.991417 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.991312    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"]
Oct 30 00:14:28.995439 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:28.995417    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.021906 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.021881    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"]
Oct 30 00:14:29.022480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.022448    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-deda90ae-openshift-default-dockercfg-z7p56\""
Oct 30 00:14:29.074178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.074149    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-envoy\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.074178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.074184    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-token\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.074412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.074221    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-socket\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.074412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.074298    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lhhm\" (UniqueName: \"kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-kube-api-access-7lhhm\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.074412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.074381    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-credential-socket\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.074542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.074483    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-data\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.074542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.074502    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istiod-ca-cert\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.074542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.074526    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-certs\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.074660 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.074548    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-podinfo\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.175443 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.175408    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-credential-socket\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.175615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.175514    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-data\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.175615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.175539    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istiod-ca-cert\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.175615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.175567    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-certs\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.175787 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.175651    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-podinfo\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.175787 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.175754    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-envoy\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.175897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.175783    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-token\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.175897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.175836    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-socket\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.175897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.175862    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lhhm\" (UniqueName: \"kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-kube-api-access-7lhhm\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.175897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.175877    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-data\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.176121 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.175860    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-credential-socket\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.176121 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.175936    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-certs\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.176448 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.176423    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-socket\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.176598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.176455    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istiod-ca-cert\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.178170 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.178144    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-envoy\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.178468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.178450    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-podinfo\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.203554 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.203525    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lhhm\" (UniqueName: \"kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-kube-api-access-7lhhm\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.204180 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.204156    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-token\") pod \"e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") " pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.325700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.325657    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:29.412788 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.412746    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"]
Oct 30 00:14:29.416193 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.416167    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.441223 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.435554    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8ffe924e-gw-openshift-default-dockercfg-xwcd5\""
Oct 30 00:14:29.441223 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.438934    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"]
Oct 30 00:14:29.482270 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.482239    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-envoy\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.482400 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.482293    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-credential-socket\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.482400 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.482316    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc927\" (UniqueName: \"kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-kube-api-access-kc927\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.482517 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.482422    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-podinfo\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.482517 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.482467    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-certs\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.482517 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.482494    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-socket\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.482676 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.482610    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e93d939d-d744-4a1b-a05c-744e65d507cb-istiod-ca-cert\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.482735 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.482718    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-data\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.482776 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.482750    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-token\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.584510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.584425    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-credential-socket\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.584510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.584476    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc927\" (UniqueName: \"kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-kube-api-access-kc927\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.584705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.584530    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-podinfo\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.584705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.584583    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-certs\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.584705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.584624    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-socket\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.584705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.584654    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e93d939d-d744-4a1b-a05c-744e65d507cb-istiod-ca-cert\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.584705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.584696    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-data\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.584909 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.584719    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-token\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.584909 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.584766    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-envoy\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.585827 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.585662    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-certs\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.585947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.585913    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-credential-socket\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.586515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.586492    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e93d939d-d744-4a1b-a05c-744e65d507cb-istiod-ca-cert\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.586616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.586555    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-data\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.586819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.586796    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-socket\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.588526 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.588503    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-podinfo\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.588874 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.588754    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-envoy\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.607960 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.607939    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc927\" (UniqueName: \"kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-kube-api-access-kc927\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.608172 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.608154    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-token\") pod \"e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") " pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.645356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.645106    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"]
Oct 30 00:14:29.649192 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:14:29.649164    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d37a88_ba16_4717_8dcd_6b6457bf4fc1.slice/crio-61874f822274fe63a95e08e7412b46433769f99e559f02d552fb939501fd4f0c WatchSource:0}: Error finding container 61874f822274fe63a95e08e7412b46433769f99e559f02d552fb939501fd4f0c: Status 404 returned error can't find the container with id 61874f822274fe63a95e08e7412b46433769f99e559f02d552fb939501fd4f0c
Oct 30 00:14:29.651588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.651559    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:29.651663 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.651622    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:29.651663 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.651653    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:29.729320 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.729290    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:29.956423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.956333    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" event={"ID":"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1","Type":"ContainerStarted","Data":"38069c0470593ba688d24a5f092d27e25f18046d4aab90ee349d55cbf018f538"}
Oct 30 00:14:29.956423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.956406    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" event={"ID":"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1","Type":"ContainerStarted","Data":"61874f822274fe63a95e08e7412b46433769f99e559f02d552fb939501fd4f0c"}
Oct 30 00:14:29.958910 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.958883    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" event={"ID":"37484057-61d0-44a9-96db-4d814905649b","Type":"ContainerStarted","Data":"e5666b8df36f3ed97ae9fb8ff8d22b413f166b52c0c196091738be96bfee841b"}
Oct 30 00:14:29.959055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:29.958935    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" event={"ID":"37484057-61d0-44a9-96db-4d814905649b","Type":"ContainerStarted","Data":"1b7f480b242b0cac038ceb48593980223c2d3708816d0e1757ba0df93c1ebbf6"}
Oct 30 00:14:30.003268 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.003196    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" podStartSLOduration=2.003177766 podStartE2EDuration="2.003177766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:30.001175317 +0000 UTC m=+579.526469216" watchObservedRunningTime="2026-10-30 00:14:30.003177766 +0000 UTC m=+579.528471645"
Oct 30 00:14:30.044971 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.044916    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" podStartSLOduration=2.04490124 podStartE2EDuration="2.04490124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:30.043865541 +0000 UTC m=+579.569159417" watchObservedRunningTime="2026-10-30 00:14:30.04490124 +0000 UTC m=+579.570195116"
Oct 30 00:14:30.056323 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.056297    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"]
Oct 30 00:14:30.058391 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:14:30.058366    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93d939d_d744_4a1b_a05c_744e65d507cb.slice/crio-453607516fceddb4ca189b9a57c19805e117752a5c8c8e13998183ccbfa2f598 WatchSource:0}: Error finding container 453607516fceddb4ca189b9a57c19805e117752a5c8c8e13998183ccbfa2f598: Status 404 returned error can't find the container with id 453607516fceddb4ca189b9a57c19805e117752a5c8c8e13998183ccbfa2f598
Oct 30 00:14:30.060701 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.060675    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:30.060772 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.060731    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:30.060772 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.060763    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:30.326625 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.326592    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:30.331257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.331235    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:30.632155 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.628844    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:30.634978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.634950    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:30.965016 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.964934    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" event={"ID":"e93d939d-d744-4a1b-a05c-744e65d507cb","Type":"ContainerStarted","Data":"2da2511da3324aa88b750875c27fb9511edccf40adf38f53a2704cab20aaed39"}
Oct 30 00:14:30.965016 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.964990    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" event={"ID":"e93d939d-d744-4a1b-a05c-744e65d507cb","Type":"ContainerStarted","Data":"453607516fceddb4ca189b9a57c19805e117752a5c8c8e13998183ccbfa2f598"}
Oct 30 00:14:30.965462 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.965442    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:30.965542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.965492    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:30.966282 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.966256    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:14:30.966486 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:30.966470    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:31.008953 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:31.008904    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" podStartSLOduration=2.00889172 podStartE2EDuration="2.00889172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:31.007129236 +0000 UTC m=+580.532423114" watchObservedRunningTime="2026-10-30 00:14:31.00889172 +0000 UTC m=+580.534185598"
Oct 30 00:14:31.730249 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:31.730220    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:31.735646 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:31.735618    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:31.971172 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:31.971131    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:31.971895 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:31.971867    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:14:33.453165 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:33.453125    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"]
Oct 30 00:14:33.460063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:33.457591    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"
Oct 30 00:14:33.474399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:33.474369    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"]
Oct 30 00:14:33.483452 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:33.483418    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-deda90ae\""
Oct 30 00:14:33.483572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:33.483450    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-deda90ae-dockercfg-xbsqb\""
Oct 30 00:14:33.524770 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:33.524731    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9k8z\" (UniqueName: \"kubernetes.io/projected/fa2ccfdf-6f75-408d-bc8e-afe656640240-kube-api-access-j9k8z\") pod \"payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs\" (UID: \"fa2ccfdf-6f75-408d-bc8e-afe656640240\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"
Oct 30 00:14:33.524969 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:33.524803    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2ccfdf-6f75-408d-bc8e-afe656640240-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs\" (UID: \"fa2ccfdf-6f75-408d-bc8e-afe656640240\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"
Oct 30 00:14:33.625522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:33.625488    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9k8z\" (UniqueName: \"kubernetes.io/projected/fa2ccfdf-6f75-408d-bc8e-afe656640240-kube-api-access-j9k8z\") pod \"payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs\" (UID: \"fa2ccfdf-6f75-408d-bc8e-afe656640240\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"
Oct 30 00:14:33.625667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:33.625528    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2ccfdf-6f75-408d-bc8e-afe656640240-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs\" (UID: \"fa2ccfdf-6f75-408d-bc8e-afe656640240\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"
Oct 30 00:14:33.626285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:33.626266    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2ccfdf-6f75-408d-bc8e-afe656640240-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs\" (UID: \"fa2ccfdf-6f75-408d-bc8e-afe656640240\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"
Oct 30 00:14:33.647800 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:33.647775    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9k8z\" (UniqueName: \"kubernetes.io/projected/fa2ccfdf-6f75-408d-bc8e-afe656640240-kube-api-access-j9k8z\") pod \"payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs\" (UID: \"fa2ccfdf-6f75-408d-bc8e-afe656640240\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"
Oct 30 00:14:33.772123 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:33.772084    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"
Oct 30 00:14:33.898939 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:14:33.898902    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:14:34.063655 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:34.063631    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"]
Oct 30 00:14:34.065518 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:14:34.065490    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2ccfdf_6f75_408d_bc8e_afe656640240.slice/crio-89004853095e8ce3b8a0f1ab31a1ca6529ebe02e47c0282cec5a931063765deb WatchSource:0}: Error finding container 89004853095e8ce3b8a0f1ab31a1ca6529ebe02e47c0282cec5a931063765deb: Status 404 returned error can't find the container with id 89004853095e8ce3b8a0f1ab31a1ca6529ebe02e47c0282cec5a931063765deb
Oct 30 00:14:34.989375 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:34.989341    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs" event={"ID":"fa2ccfdf-6f75-408d-bc8e-afe656640240","Type":"ContainerStarted","Data":"89004853095e8ce3b8a0f1ab31a1ca6529ebe02e47c0282cec5a931063765deb"}
Oct 30 00:14:35.164633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.164596    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"]
Oct 30 00:14:35.167607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.167579    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"
Oct 30 00:14:35.186997 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.186971    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"]
Oct 30 00:14:35.196064 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.196012    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6c9e9fa7\""
Oct 30 00:14:35.196178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.196011    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6c9e9fa7-dockercfg-6vxrv\""
Oct 30 00:14:35.245128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.242352    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8bg\" (UniqueName: \"kubernetes.io/projected/db85767a-29f1-4c0b-a912-3e4c1af1cd83-kube-api-access-nj8bg\") pod \"payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr\" (UID: \"db85767a-29f1-4c0b-a912-3e4c1af1cd83\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"
Oct 30 00:14:35.245128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.242439    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db85767a-29f1-4c0b-a912-3e4c1af1cd83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr\" (UID: \"db85767a-29f1-4c0b-a912-3e4c1af1cd83\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"
Oct 30 00:14:35.256598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.256555    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"]
Oct 30 00:14:35.278082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.276076    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"
Oct 30 00:14:35.278082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.277118    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"]
Oct 30 00:14:35.343278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.343244    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm269\" (UniqueName: \"kubernetes.io/projected/95e5b2df-6caa-4421-9a87-492979195236-kube-api-access-xm269\") pod \"payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd\" (UID: \"95e5b2df-6caa-4421-9a87-492979195236\") " pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"
Oct 30 00:14:35.343446 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.343291    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8bg\" (UniqueName: \"kubernetes.io/projected/db85767a-29f1-4c0b-a912-3e4c1af1cd83-kube-api-access-nj8bg\") pod \"payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr\" (UID: \"db85767a-29f1-4c0b-a912-3e4c1af1cd83\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"
Oct 30 00:14:35.343446 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.343396    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db85767a-29f1-4c0b-a912-3e4c1af1cd83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr\" (UID: \"db85767a-29f1-4c0b-a912-3e4c1af1cd83\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"
Oct 30 00:14:35.343581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.343460    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e5b2df-6caa-4421-9a87-492979195236-plugins-config-volume\") pod \"payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd\" (UID: \"95e5b2df-6caa-4421-9a87-492979195236\") " pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"
Oct 30 00:14:35.344279 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.344252    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db85767a-29f1-4c0b-a912-3e4c1af1cd83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr\" (UID: \"db85767a-29f1-4c0b-a912-3e4c1af1cd83\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"
Oct 30 00:14:35.367334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.367310    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8bg\" (UniqueName: \"kubernetes.io/projected/db85767a-29f1-4c0b-a912-3e4c1af1cd83-kube-api-access-nj8bg\") pod \"payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr\" (UID: \"db85767a-29f1-4c0b-a912-3e4c1af1cd83\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"
Oct 30 00:14:35.444103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.444063    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm269\" (UniqueName: \"kubernetes.io/projected/95e5b2df-6caa-4421-9a87-492979195236-kube-api-access-xm269\") pod \"payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd\" (UID: \"95e5b2df-6caa-4421-9a87-492979195236\") " pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"
Oct 30 00:14:35.444260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.444176    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e5b2df-6caa-4421-9a87-492979195236-plugins-config-volume\") pod \"payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd\" (UID: \"95e5b2df-6caa-4421-9a87-492979195236\") " pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"
Oct 30 00:14:35.445296 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.444912    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e5b2df-6caa-4421-9a87-492979195236-plugins-config-volume\") pod \"payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd\" (UID: \"95e5b2df-6caa-4421-9a87-492979195236\") " pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"
Oct 30 00:14:35.469977 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.469942    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm269\" (UniqueName: \"kubernetes.io/projected/95e5b2df-6caa-4421-9a87-492979195236-kube-api-access-xm269\") pod \"payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd\" (UID: \"95e5b2df-6caa-4421-9a87-492979195236\") " pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"
Oct 30 00:14:35.483786 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.483758    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"
Oct 30 00:14:35.600436 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.600406    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"
Oct 30 00:14:35.804505 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:14:35.804465    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb85767a_29f1_4c0b_a912_3e4c1af1cd83.slice/crio-1613d7026dd0df794c7a0396cb452ab21f7d3d40fb38eeaad846a94e640c3a6f WatchSource:0}: Error finding container 1613d7026dd0df794c7a0396cb452ab21f7d3d40fb38eeaad846a94e640c3a6f: Status 404 returned error can't find the container with id 1613d7026dd0df794c7a0396cb452ab21f7d3d40fb38eeaad846a94e640c3a6f
Oct 30 00:14:35.807576 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.807484    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"]
Oct 30 00:14:35.910469 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.910446    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"]
Oct 30 00:14:35.913233 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:14:35.913205    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e5b2df_6caa_4421_9a87_492979195236.slice/crio-f3f93672460889415fbb70c1ffe3a35269d68faaefe1739f38a0ea4fb796f632 WatchSource:0}: Error finding container f3f93672460889415fbb70c1ffe3a35269d68faaefe1739f38a0ea4fb796f632: Status 404 returned error can't find the container with id f3f93672460889415fbb70c1ffe3a35269d68faaefe1739f38a0ea4fb796f632
Oct 30 00:14:35.996364 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.996330    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd" event={"ID":"95e5b2df-6caa-4421-9a87-492979195236","Type":"ContainerStarted","Data":"f3f93672460889415fbb70c1ffe3a35269d68faaefe1739f38a0ea4fb796f632"}
Oct 30 00:14:35.997798 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:35.997771    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr" event={"ID":"db85767a-29f1-4c0b-a912-3e4c1af1cd83","Type":"ContainerStarted","Data":"1613d7026dd0df794c7a0396cb452ab21f7d3d40fb38eeaad846a94e640c3a6f"}
Oct 30 00:14:36.011213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:36.011184    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"]
Oct 30 00:14:36.016807 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:36.016767    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"
Oct 30 00:14:36.030960 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:36.030937    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-185595da-dockercfg-wv4lp\""
Oct 30 00:14:36.032893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:36.032863    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"]
Oct 30 00:14:36.041467 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:36.041436    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-185595da\""
Oct 30 00:14:36.054547 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:36.054251    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwg92\" (UniqueName: \"kubernetes.io/projected/c9e4be2a-91f9-48c9-9d2a-630a6201811b-kube-api-access-vwg92\") pod \"payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq\" (UID: \"c9e4be2a-91f9-48c9-9d2a-630a6201811b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"
Oct 30 00:14:36.054547 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:36.054315    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e4be2a-91f9-48c9-9d2a-630a6201811b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq\" (UID: \"c9e4be2a-91f9-48c9-9d2a-630a6201811b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"
Oct 30 00:14:36.170772 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:36.170729    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwg92\" (UniqueName: \"kubernetes.io/projected/c9e4be2a-91f9-48c9-9d2a-630a6201811b-kube-api-access-vwg92\") pod \"payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq\" (UID: \"c9e4be2a-91f9-48c9-9d2a-630a6201811b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"
Oct 30 00:14:36.170958 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:36.170791    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e4be2a-91f9-48c9-9d2a-630a6201811b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq\" (UID: \"c9e4be2a-91f9-48c9-9d2a-630a6201811b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"
Oct 30 00:14:36.171782 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:36.171758    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e4be2a-91f9-48c9-9d2a-630a6201811b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq\" (UID: \"c9e4be2a-91f9-48c9-9d2a-630a6201811b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"
Oct 30 00:14:36.201507 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:36.201470    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwg92\" (UniqueName: \"kubernetes.io/projected/c9e4be2a-91f9-48c9-9d2a-630a6201811b-kube-api-access-vwg92\") pod \"payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq\" (UID: \"c9e4be2a-91f9-48c9-9d2a-630a6201811b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"
Oct 30 00:14:36.331581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:36.331557    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"
Oct 30 00:14:36.632555 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:36.632458    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"]
Oct 30 00:14:36.636688 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:14:36.636663    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e4be2a_91f9_48c9_9d2a_630a6201811b.slice/crio-a6a40ea1b2e40909ea98fa96a2259f0cf043eaecdcf0be992468c989f9ef2070 WatchSource:0}: Error finding container a6a40ea1b2e40909ea98fa96a2259f0cf043eaecdcf0be992468c989f9ef2070: Status 404 returned error can't find the container with id a6a40ea1b2e40909ea98fa96a2259f0cf043eaecdcf0be992468c989f9ef2070
Oct 30 00:14:37.004275 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.004221    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr" event={"ID":"db85767a-29f1-4c0b-a912-3e4c1af1cd83","Type":"ContainerStarted","Data":"635ed80b1b8b4240327f641b70b7aa863af8417395d32bf47985b118d7e8c1de"}
Oct 30 00:14:37.004727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.004350    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"
Oct 30 00:14:37.005729 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.005706    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq" event={"ID":"c9e4be2a-91f9-48c9-9d2a-630a6201811b","Type":"ContainerStarted","Data":"37482fb1f175584e5fec90265ab575e5e761bb205ded4b47eec59dbb0b6f15ae"}
Oct 30 00:14:37.005814 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.005746    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"
Oct 30 00:14:37.005814 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.005761    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq" event={"ID":"c9e4be2a-91f9-48c9-9d2a-630a6201811b","Type":"ContainerStarted","Data":"a6a40ea1b2e40909ea98fa96a2259f0cf043eaecdcf0be992468c989f9ef2070"}
Oct 30 00:14:37.007231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.007211    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd" event={"ID":"95e5b2df-6caa-4421-9a87-492979195236","Type":"ContainerStarted","Data":"5f507c881e2bb648419df8b3334c6ade0dd861e1960beda4a103522961dac69a"}
Oct 30 00:14:37.007326 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.007262    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"
Oct 30 00:14:37.008554 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.008530    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs" event={"ID":"fa2ccfdf-6f75-408d-bc8e-afe656640240","Type":"ContainerStarted","Data":"8842e283d6d0b55a74e07209af8c475eabff326a88b79b56aca14d5c771db48f"}
Oct 30 00:14:37.008672 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.008658    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"
Oct 30 00:14:37.051397 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.051339    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr" podStartSLOduration=1.527439136 podStartE2EDuration="2.051327264s" totalImagesPullingTime="523.888128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:37.049090369 +0000 UTC m=+586.574384249" watchObservedRunningTime="2026-10-30 00:14:37.051327264 +0000 UTC m=+586.576621140"
Oct 30 00:14:37.088734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.088693    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq" podStartSLOduration=2.08868242 podStartE2EDuration="2.08868242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:37.087017264 +0000 UTC m=+586.612311142" watchObservedRunningTime="2026-10-30 00:14:37.08868242 +0000 UTC m=+586.613976296"
Oct 30 00:14:37.145687 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.145650    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs" podStartSLOduration=1.884102028 podStartE2EDuration="4.145639262s" totalImagesPullingTime="2.261537234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:37.144853381 +0000 UTC m=+586.670147258" watchObservedRunningTime="2026-10-30 00:14:37.145639262 +0000 UTC m=+586.670933138"
Oct 30 00:14:37.244166 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.244130    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd" podStartSLOduration=1.829168992 podStartE2EDuration="2.244118265s" totalImagesPullingTime="414.949273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:37.242680007 +0000 UTC m=+586.767973887" watchObservedRunningTime="2026-10-30 00:14:37.244118265 +0000 UTC m=+586.769412142"
Oct 30 00:14:37.925387 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.925357    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"]
Oct 30 00:14:37.929150 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.929121    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:37.952117 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.952092    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"]
Oct 30 00:14:37.960258 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.960239    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7a7ac6-openshift-default-dockercfg-rv5lp\""
Oct 30 00:14:37.990224 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.990198    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:37.990315 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.990264    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hps5z\" (UniqueName: \"kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-kube-api-access-hps5z\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:37.990315 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.990309    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:37.990394 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.990333    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:37.990441 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.990421    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:37.990485 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.990472    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:37.990561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.990541    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:37.990593 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.990582    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:37.990622 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:37.990605    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.092363    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.092531    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.092623    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hps5z\" (UniqueName: \"kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-kube-api-access-hps5z\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.092702    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.092742    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.092799    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.093603    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.093949    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.094614    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.094712    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.094885    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.094922    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.094965    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.095664    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.107058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.101408    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.122052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.118777    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.123832 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.123773    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.132324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.132276    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hps5z\" (UniqueName: \"kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-kube-api-access-hps5z\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.243695 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.243665    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:38.530542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.530469    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"]
Oct 30 00:14:38.534757 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.534734    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"
Oct 30 00:14:38.550665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.550638    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-6c45bb-dockercfg-d8k6b\""
Oct 30 00:14:38.553523 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.553495    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"]
Oct 30 00:14:38.573150 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.573120    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"]
Oct 30 00:14:38.582084 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:14:38.582059    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f550b5f_f099_42d9_bd19_66ec69fce6d5.slice/crio-ec341f294515735be00bfd5973e677acfea37b9399dd9b0d3de5c41135308119 WatchSource:0}: Error finding container ec341f294515735be00bfd5973e677acfea37b9399dd9b0d3de5c41135308119: Status 404 returned error can't find the container with id ec341f294515735be00bfd5973e677acfea37b9399dd9b0d3de5c41135308119
Oct 30 00:14:38.584684 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.584650    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:38.584783 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.584728    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:38.584783 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.584767    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:38.594763 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.594730    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-6c45bb\""
Oct 30 00:14:38.600163 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.600143    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85k9j\" (UniqueName: \"kubernetes.io/projected/35394583-9388-440a-8cc8-e0170d541e36-kube-api-access-85k9j\") pod \"payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh\" (UID: \"35394583-9388-440a-8cc8-e0170d541e36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"
Oct 30 00:14:38.600269 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.600251    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35394583-9388-440a-8cc8-e0170d541e36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh\" (UID: \"35394583-9388-440a-8cc8-e0170d541e36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"
Oct 30 00:14:38.701095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.701065    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85k9j\" (UniqueName: \"kubernetes.io/projected/35394583-9388-440a-8cc8-e0170d541e36-kube-api-access-85k9j\") pod \"payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh\" (UID: \"35394583-9388-440a-8cc8-e0170d541e36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"
Oct 30 00:14:38.701219 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.701161    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35394583-9388-440a-8cc8-e0170d541e36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh\" (UID: \"35394583-9388-440a-8cc8-e0170d541e36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"
Oct 30 00:14:38.702224 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.702195    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35394583-9388-440a-8cc8-e0170d541e36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh\" (UID: \"35394583-9388-440a-8cc8-e0170d541e36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"
Oct 30 00:14:38.723448 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.723420    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85k9j\" (UniqueName: \"kubernetes.io/projected/35394583-9388-440a-8cc8-e0170d541e36-kube-api-access-85k9j\") pod \"payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh\" (UID: \"35394583-9388-440a-8cc8-e0170d541e36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"
Oct 30 00:14:38.847837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:38.847748    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"
Oct 30 00:14:39.019798 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.019762    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" event={"ID":"1f550b5f-f099-42d9-bd19-66ec69fce6d5","Type":"ContainerStarted","Data":"5598768fe2770195b6b21bb83e54cde9c93eb45f006f47fe6486c49f975d280d"}
Oct 30 00:14:39.019954 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.019848    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" event={"ID":"1f550b5f-f099-42d9-bd19-66ec69fce6d5","Type":"ContainerStarted","Data":"ec341f294515735be00bfd5973e677acfea37b9399dd9b0d3de5c41135308119"}
Oct 30 00:14:39.063651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.063591    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" podStartSLOduration=2.063576963 podStartE2EDuration="2.063576963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:39.061784588 +0000 UTC m=+588.587078465" watchObservedRunningTime="2026-10-30 00:14:39.063576963 +0000 UTC m=+588.588870882"
Oct 30 00:14:39.144930 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.144899    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"]
Oct 30 00:14:39.147466 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:14:39.147439    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35394583_9388_440a_8cc8_e0170d541e36.slice/crio-f6b7a47d2cd90bd2c87f48d8154d6f4713c3037ca8bb3e7c69a7e1e142fcdcd5 WatchSource:0}: Error finding container f6b7a47d2cd90bd2c87f48d8154d6f4713c3037ca8bb3e7c69a7e1e142fcdcd5: Status 404 returned error can't find the container with id f6b7a47d2cd90bd2c87f48d8154d6f4713c3037ca8bb3e7c69a7e1e142fcdcd5
Oct 30 00:14:39.244292 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.244226    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:39.249551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.249525    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:39.440302 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.440212    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"]
Oct 30 00:14:39.443625 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.443599    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"
Oct 30 00:14:39.458722 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.458690    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8ffe924e-dockercfg-hnzps\""
Oct 30 00:14:39.458901 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.458878    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8ffe924e\""
Oct 30 00:14:39.477067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.477012    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"]
Oct 30 00:14:39.524069 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.518988    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:14:39.609990 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.609956    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdvnx\" (UniqueName: \"kubernetes.io/projected/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-kube-api-access-xdvnx\") pod \"payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl\" (UID: \"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"
Oct 30 00:14:39.610167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.610105    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl\" (UID: \"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"
Oct 30 00:14:39.712132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.711785    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl\" (UID: \"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"
Oct 30 00:14:39.712132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.711919    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdvnx\" (UniqueName: \"kubernetes.io/projected/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-kube-api-access-xdvnx\") pod \"payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl\" (UID: \"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"
Oct 30 00:14:39.714285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.714257    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl\" (UID: \"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"
Oct 30 00:14:39.741703 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.741675    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdvnx\" (UniqueName: \"kubernetes.io/projected/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-kube-api-access-xdvnx\") pod \"payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl\" (UID: \"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"
Oct 30 00:14:39.756644 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:39.756621    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"
Oct 30 00:14:40.026767 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:40.026727    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh" event={"ID":"35394583-9388-440a-8cc8-e0170d541e36","Type":"ContainerStarted","Data":"e6f6fb96bd259102e37dd8dc3c8d1bce54c357c133206441f0ac4dbcacebdf10"}
Oct 30 00:14:40.026988 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:40.026787    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh" event={"ID":"35394583-9388-440a-8cc8-e0170d541e36","Type":"ContainerStarted","Data":"f6b7a47d2cd90bd2c87f48d8154d6f4713c3037ca8bb3e7c69a7e1e142fcdcd5"}
Oct 30 00:14:40.027134 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:40.027115    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"
Oct 30 00:14:40.027200 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:40.027163    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:40.028356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:40.028336    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:14:40.068559 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:40.068515    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh" podStartSLOduration=2.068501816 podStartE2EDuration="2.068501816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:40.065825433 +0000 UTC m=+589.591119311" watchObservedRunningTime="2026-10-30 00:14:40.068501816 +0000 UTC m=+589.593795738"
Oct 30 00:14:40.083643 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:40.083617    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"]
Oct 30 00:14:40.086918 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:14:40.086894    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fa9de1_11bd_49ea_9d8d_0ce5f193a621.slice/crio-8b778f9e837b58f7992951fdac1341c517cf07927c9ada85825e8c86a9e7641a WatchSource:0}: Error finding container 8b778f9e837b58f7992951fdac1341c517cf07927c9ada85825e8c86a9e7641a: Status 404 returned error can't find the container with id 8b778f9e837b58f7992951fdac1341c517cf07927c9ada85825e8c86a9e7641a
Oct 30 00:14:41.056337 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:41.056280    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl" event={"ID":"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621","Type":"ContainerStarted","Data":"f169d6c772229a3cc2f6e0584e6e8a29eaa703b10ce35b9a082661c4c2266f4e"}
Oct 30 00:14:41.056934 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:41.056886    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl" event={"ID":"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621","Type":"ContainerStarted","Data":"8b778f9e837b58f7992951fdac1341c517cf07927c9ada85825e8c86a9e7641a"}
Oct 30 00:14:41.057188 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:41.057151    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"
Oct 30 00:14:41.096796 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:41.096748    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl" podStartSLOduration=2.096733511 podStartE2EDuration="2.096733511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:41.095360852 +0000 UTC m=+590.620654728" watchObservedRunningTime="2026-10-30 00:14:41.096733511 +0000 UTC m=+590.622027387"
Oct 30 00:14:43.685652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.685614    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"]
Oct 30 00:14:43.689109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.689083    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:43.705596 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.705554    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8ffe924e-serving-cert\""
Oct 30 00:14:43.719080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.717529    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:14:43.719080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.717600    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:14:43.719080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.717603    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 00:14:43.719080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.717744    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w2hx9\""
Oct 30 00:14:43.728049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.721300    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"]
Oct 30 00:14:43.785109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.785064    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q4hr\" (UniqueName: \"kubernetes.io/projected/1994b6ff-5502-487b-8ca0-33d53ac8d41b-kube-api-access-5q4hr\") pod \"maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:43.785109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.785109    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-metrics-certs\") pod \"maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:43.785313 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.785213    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-maas-api-tls\") pod \"maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:43.785313 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.785234    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1994b6ff-5502-487b-8ca0-33d53ac8d41b-openshift-service-ca\") pod \"maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:43.886672 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.886636    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-maas-api-tls\") pod \"maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:43.886859 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.886682    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1994b6ff-5502-487b-8ca0-33d53ac8d41b-openshift-service-ca\") pod \"maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:43.886859 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.886822    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q4hr\" (UniqueName: \"kubernetes.io/projected/1994b6ff-5502-487b-8ca0-33d53ac8d41b-kube-api-access-5q4hr\") pod \"maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:43.886859 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.886858    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-metrics-certs\") pod \"maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:43.887852 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.887799    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1994b6ff-5502-487b-8ca0-33d53ac8d41b-openshift-service-ca\") pod \"maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:43.889749 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.889725    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-metrics-certs\") pod \"maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:43.890183 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.890159    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-maas-api-tls\") pod \"maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:43.914064 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:43.914002    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q4hr\" (UniqueName: \"kubernetes.io/projected/1994b6ff-5502-487b-8ca0-33d53ac8d41b-kube-api-access-5q4hr\") pod \"maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:44.002216 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:44.002187    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:44.374342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:44.374316    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"]
Oct 30 00:14:44.376419 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:14:44.376388    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1994b6ff_5502_487b_8ca0_33d53ac8d41b.slice/crio-f7b24e514f049d5fcb18a9685707f1cf4e7db8b73e4f5aff1bf1b7c0857a0cea WatchSource:0}: Error finding container f7b24e514f049d5fcb18a9685707f1cf4e7db8b73e4f5aff1bf1b7c0857a0cea: Status 404 returned error can't find the container with id f7b24e514f049d5fcb18a9685707f1cf4e7db8b73e4f5aff1bf1b7c0857a0cea
Oct 30 00:14:45.022050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:45.003472    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:14:45.087127 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:45.087038    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w" event={"ID":"1994b6ff-5502-487b-8ca0-33d53ac8d41b","Type":"ContainerStarted","Data":"f7b24e514f049d5fcb18a9685707f1cf4e7db8b73e4f5aff1bf1b7c0857a0cea"}
Oct 30 00:14:48.017713 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:48.017682    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"
Oct 30 00:14:48.018288 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:48.017758    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"
Oct 30 00:14:48.018288 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:48.018108    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"
Oct 30 00:14:48.018981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:48.018960    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"
Oct 30 00:14:49.118677 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:49.118645    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w" event={"ID":"1994b6ff-5502-487b-8ca0-33d53ac8d41b","Type":"ContainerStarted","Data":"880e5ceda44086781d4e165e02b8afeb8def027dafc75348e9ed04a7c527e356"}
Oct 30 00:14:49.119101 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:49.118737    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:49.160184 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:49.160124    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w" podStartSLOduration=2.486879761 podStartE2EDuration="6.160105598s" totalImagesPullingTime="3.673225837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:49.158167885 +0000 UTC m=+598.683461762" watchObservedRunningTime="2026-10-30 00:14:49.160105598 +0000 UTC m=+598.685399477"
Oct 30 00:14:50.126255 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:50.126180    4441 generic.go:356] "Generic (PLEG): container finished" podID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerID="38069c0470593ba688d24a5f092d27e25f18046d4aab90ee349d55cbf018f538" exitCode=0
Oct 30 00:14:50.126729 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:50.126258    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" event={"ID":"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1","Type":"ContainerDied","Data":"38069c0470593ba688d24a5f092d27e25f18046d4aab90ee349d55cbf018f538"}
Oct 30 00:14:50.126729 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:50.126664    4441 scope.go:122] "RemoveContainer" containerID="38069c0470593ba688d24a5f092d27e25f18046d4aab90ee349d55cbf018f538"
Oct 30 00:14:50.128151 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:50.128123    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:50.128222 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:50.128206    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:50.128259 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:50.128250    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:14:50.772936 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:50.772906    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:14:50.779712 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:50.779688    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:14:51.074287 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:51.074255    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"
Oct 30 00:14:51.227050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:51.211580    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" event={"ID":"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1","Type":"ContainerStarted","Data":"7082d1426c878647ea9055efabd6bfbb11d271f9cb878a4c927a3243b6ba92c4"}
Oct 30 00:14:51.342695 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:51.342610    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:51.354055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:51.352141    4441 patch_prober.go:29] interesting pod/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 00:14:51.354055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:51.352203    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 00:14:52.326418 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:52.326385    4441 patch_prober.go:29] interesting pod/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 00:14:52.335136 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:52.335097    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 00:14:53.173050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:53.172121    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"
Oct 30 00:14:53.331928 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:53.329386    4441 patch_prober.go:29] interesting pod/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 00:14:53.331928 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:53.329446    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 00:14:54.341517 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.341074    4441 patch_prober.go:29] interesting pod/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 00:14:54.341517 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.341133    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 00:14:54.805781 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.805743    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"]
Oct 30 00:14:54.840045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.833901    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"]
Oct 30 00:14:54.840045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.834138    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:54.840045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.835676    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:54.840045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.835762    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-metrics-certs\") pod \"maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:54.840045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.835836    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcxpj\" (UniqueName: \"kubernetes.io/projected/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-kube-api-access-rcxpj\") pod \"maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:54.840045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.835884    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:54.852250 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.852227    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-7a7ac6-serving-cert\""
Oct 30 00:14:54.953050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.948913    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcxpj\" (UniqueName: \"kubernetes.io/projected/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-kube-api-access-rcxpj\") pod \"maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:54.953050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.948985    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:54.953050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.949078    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:54.953050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.949147    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-metrics-certs\") pod \"maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:54.958254 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:14:54.958214    4441 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-serving-cert: secret "maas-api-e2e-worker-w1-7a7ac6-serving-cert" not found
Oct 30 00:14:54.964053 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:14:54.958639    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-maas-api-tls podName:b4977042-bf8c-4bc6-8462-7f84acf9e5a2 nodeName:}" failed. No retries permitted until 2026-10-30 00:14:55.458505549 +0000 UTC m=+604.983799411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-maas-api-tls") pod "maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8" (UID: "b4977042-bf8c-4bc6-8462-7f84acf9e5a2") : secret "maas-api-e2e-worker-w1-7a7ac6-serving-cert" not found
Oct 30 00:14:54.980100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:54.973048    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:55.023082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:55.019824    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcxpj\" (UniqueName: \"kubernetes.io/projected/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-kube-api-access-rcxpj\") pod \"maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:55.100103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:55.093696    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-metrics-certs\") pod \"maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:55.202339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:55.201227    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:14:55.343049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:55.335487    4441 patch_prober.go:29] interesting pod/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 00:14:55.343049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:55.335544    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 00:14:55.513049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:55.512819    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:55.516809 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:55.516783    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:55.553050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:55.548326    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:56.343050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:56.338116    4441 patch_prober.go:29] interesting pod/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 00:14:56.343050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:56.338213    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 00:14:56.506219 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:56.506064    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"]
Oct 30 00:14:56.509430 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:14:56.509298    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4977042_bf8c_4bc6_8462_7f84acf9e5a2.slice/crio-68adc4484d2cc2fde4cc9710b9a1bc5ee8d73835147f68af4486530eb2961620 WatchSource:0}: Error finding container 68adc4484d2cc2fde4cc9710b9a1bc5ee8d73835147f68af4486530eb2961620: Status 404 returned error can't find the container with id 68adc4484d2cc2fde4cc9710b9a1bc5ee8d73835147f68af4486530eb2961620
Oct 30 00:14:57.332119 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:57.332089    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:57.333215 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:57.333186    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:57.333817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:57.333794    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:14:57.510648 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:57.510612    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8" event={"ID":"b4977042-bf8c-4bc6-8462-7f84acf9e5a2","Type":"ContainerStarted","Data":"128a15f0f851ca3483c759b4528263602b231c9d58f6149e0c764f4facf1f484"}
Oct 30 00:14:57.511189 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:57.511165    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8" event={"ID":"b4977042-bf8c-4bc6-8462-7f84acf9e5a2","Type":"ContainerStarted","Data":"68adc4484d2cc2fde4cc9710b9a1bc5ee8d73835147f68af4486530eb2961620"}
Oct 30 00:14:57.511324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:57.511311    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:14:57.559705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:57.558158    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8" podStartSLOduration=3.215370449 podStartE2EDuration="3.558136017s" totalImagesPullingTime="342.765568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:57.551324319 +0000 UTC m=+607.076618197" watchObservedRunningTime="2026-10-30 00:14:57.558136017 +0000 UTC m=+607.083429892"
Oct 30 00:14:59.170050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:14:59.157229    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:15:00.155714 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.154840    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"]
Oct 30 00:15:00.199462 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.199419    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"]
Oct 30 00:15:00.200095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.200071    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"
Oct 30 00:15:00.276188 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.270102    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\""
Oct 30 00:15:00.277849 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.277776    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"]
Oct 30 00:15:00.278202 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.278103    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"
Oct 30 00:15:00.296636 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.296611    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"]
Oct 30 00:15:00.296750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.296661    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"]
Oct 30 00:15:00.296750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.296690    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"]
Oct 30 00:15:00.296866 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.296800    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"
Oct 30 00:15:00.308201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.308182    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"]
Oct 30 00:15:00.308363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.308348    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"]
Oct 30 00:15:00.308497 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.308287    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"
Oct 30 00:15:00.369286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.367215    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8\" (UID: \"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"
Oct 30 00:15:00.369286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.367280    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxxh8\" (UniqueName: \"kubernetes.io/projected/de3547ac-b71e-4dd3-9ccc-ece14a199576-kube-api-access-qxxh8\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d\" (UID: \"de3547ac-b71e-4dd3-9ccc-ece14a199576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"
Oct 30 00:15:00.369286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.367314    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm7vg\" (UniqueName: \"kubernetes.io/projected/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-kube-api-access-rm7vg\") pod \"maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8\" (UID: \"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"
Oct 30 00:15:00.369286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.367348    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f451d6-9227-4ed4-9301-3a2565e4e479-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888655-bxdjz\" (UID: \"92f451d6-9227-4ed4-9301-3a2565e4e479\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"
Oct 30 00:15:00.369286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.367408    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sj27\" (UniqueName: \"kubernetes.io/projected/92f451d6-9227-4ed4-9301-3a2565e4e479-kube-api-access-8sj27\") pod \"maas-api-key-cleanup-29888655-bxdjz\" (UID: \"92f451d6-9227-4ed4-9301-3a2565e4e479\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"
Oct 30 00:15:00.369286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.367460    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqmzc\" (UniqueName: \"kubernetes.io/projected/64908684-4f66-47fa-b0a8-bc52b38ce34d-kube-api-access-rqmzc\") pod \"maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9\" (UID: \"64908684-4f66-47fa-b0a8-bc52b38ce34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"
Oct 30 00:15:00.369286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.367495    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64908684-4f66-47fa-b0a8-bc52b38ce34d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9\" (UID: \"64908684-4f66-47fa-b0a8-bc52b38ce34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"
Oct 30 00:15:00.369286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.367528    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de3547ac-b71e-4dd3-9ccc-ece14a199576-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d\" (UID: \"de3547ac-b71e-4dd3-9ccc-ece14a199576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"
Oct 30 00:15:00.468561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.468482    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7vg\" (UniqueName: \"kubernetes.io/projected/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-kube-api-access-rm7vg\") pod \"maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8\" (UID: \"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"
Oct 30 00:15:00.468561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.468521    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f451d6-9227-4ed4-9301-3a2565e4e479-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888655-bxdjz\" (UID: \"92f451d6-9227-4ed4-9301-3a2565e4e479\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"
Oct 30 00:15:00.468561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.468560    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sj27\" (UniqueName: \"kubernetes.io/projected/92f451d6-9227-4ed4-9301-3a2565e4e479-kube-api-access-8sj27\") pod \"maas-api-key-cleanup-29888655-bxdjz\" (UID: \"92f451d6-9227-4ed4-9301-3a2565e4e479\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"
Oct 30 00:15:00.468867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.468605    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqmzc\" (UniqueName: \"kubernetes.io/projected/64908684-4f66-47fa-b0a8-bc52b38ce34d-kube-api-access-rqmzc\") pod \"maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9\" (UID: \"64908684-4f66-47fa-b0a8-bc52b38ce34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"
Oct 30 00:15:00.468867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.468638    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64908684-4f66-47fa-b0a8-bc52b38ce34d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9\" (UID: \"64908684-4f66-47fa-b0a8-bc52b38ce34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"
Oct 30 00:15:00.468867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.468673    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de3547ac-b71e-4dd3-9ccc-ece14a199576-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d\" (UID: \"de3547ac-b71e-4dd3-9ccc-ece14a199576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"
Oct 30 00:15:00.468867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.468764    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8\" (UID: \"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"
Oct 30 00:15:00.468867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.468815    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxxh8\" (UniqueName: \"kubernetes.io/projected/de3547ac-b71e-4dd3-9ccc-ece14a199576-kube-api-access-qxxh8\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d\" (UID: \"de3547ac-b71e-4dd3-9ccc-ece14a199576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"
Oct 30 00:15:00.469364 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.469340    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f451d6-9227-4ed4-9301-3a2565e4e479-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888655-bxdjz\" (UID: \"92f451d6-9227-4ed4-9301-3a2565e4e479\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"
Oct 30 00:15:00.469571 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.469552    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de3547ac-b71e-4dd3-9ccc-ece14a199576-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d\" (UID: \"de3547ac-b71e-4dd3-9ccc-ece14a199576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"
Oct 30 00:15:00.469781 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.469758    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8\" (UID: \"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"
Oct 30 00:15:00.469973 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.469955    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64908684-4f66-47fa-b0a8-bc52b38ce34d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9\" (UID: \"64908684-4f66-47fa-b0a8-bc52b38ce34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"
Oct 30 00:15:00.491309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.491287    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqmzc\" (UniqueName: \"kubernetes.io/projected/64908684-4f66-47fa-b0a8-bc52b38ce34d-kube-api-access-rqmzc\") pod \"maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9\" (UID: \"64908684-4f66-47fa-b0a8-bc52b38ce34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"
Oct 30 00:15:00.491444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.491382    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxxh8\" (UniqueName: \"kubernetes.io/projected/de3547ac-b71e-4dd3-9ccc-ece14a199576-kube-api-access-qxxh8\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d\" (UID: \"de3547ac-b71e-4dd3-9ccc-ece14a199576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"
Oct 30 00:15:00.491634 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.491614    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm7vg\" (UniqueName: \"kubernetes.io/projected/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-kube-api-access-rm7vg\") pod \"maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8\" (UID: \"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"
Oct 30 00:15:00.491869 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.491855    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sj27\" (UniqueName: \"kubernetes.io/projected/92f451d6-9227-4ed4-9301-3a2565e4e479-kube-api-access-8sj27\") pod \"maas-api-key-cleanup-29888655-bxdjz\" (UID: \"92f451d6-9227-4ed4-9301-3a2565e4e479\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"
Oct 30 00:15:00.548049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.546646    4441 generic.go:356] "Generic (PLEG): container finished" podID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerID="2da2511da3324aa88b750875c27fb9511edccf40adf38f53a2704cab20aaed39" exitCode=0
Oct 30 00:15:00.548049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.546719    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" event={"ID":"e93d939d-d744-4a1b-a05c-744e65d507cb","Type":"ContainerDied","Data":"2da2511da3324aa88b750875c27fb9511edccf40adf38f53a2704cab20aaed39"}
Oct 30 00:15:00.548049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.547179    4441 scope.go:122] "RemoveContainer" containerID="2da2511da3324aa88b750875c27fb9511edccf40adf38f53a2704cab20aaed39"
Oct 30 00:15:00.552044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.549106    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:00.552044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.549196    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:00.552044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.549244    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:00.588538 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.588510    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"
Oct 30 00:15:00.605128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.602039    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"
Oct 30 00:15:00.663143 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.658056    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"
Oct 30 00:15:00.686252 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:00.686220    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"
Oct 30 00:15:01.067699 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.067576    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"]
Oct 30 00:15:01.070742 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.070659    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"]
Oct 30 00:15:01.083712 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:15:01.077844    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536c3f1d_9ca9_4bcf_bed5_d63c0affe07a.slice/crio-73a38905b2c71ac41b0944359ce6cbc142854fed0b2fefea2e76a260325696e5 WatchSource:0}: Error finding container 73a38905b2c71ac41b0944359ce6cbc142854fed0b2fefea2e76a260325696e5: Status 404 returned error can't find the container with id 73a38905b2c71ac41b0944359ce6cbc142854fed0b2fefea2e76a260325696e5
Oct 30 00:15:01.083712 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.080461    4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:15:01.243527 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.220976    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": read tcp 10.133.0.2:39068->10.133.0.36:15021: read: connection reset by peer" start-of-body=
Oct 30 00:15:01.243527 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.237139    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": read tcp 10.133.0.2:39068->10.133.0.36:15021: read: connection reset by peer"
Oct 30 00:15:01.243527 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.229908    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"]
Oct 30 00:15:01.263511 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.263486    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"]
Oct 30 00:15:01.281847 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:15:01.281815    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3547ac_b71e_4dd3_9ccc_ece14a199576.slice/crio-8d49c34edd3cf3e9e5a0de4aa53d08bda829970fc9d49f917c72cd2f7a2db344 WatchSource:0}: Error finding container 8d49c34edd3cf3e9e5a0de4aa53d08bda829970fc9d49f917c72cd2f7a2db344: Status 404 returned error can't find the container with id 8d49c34edd3cf3e9e5a0de4aa53d08bda829970fc9d49f917c72cd2f7a2db344
Oct 30 00:15:01.289121 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:15:01.289096    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f451d6_9227_4ed4_9301_3a2565e4e479.slice/crio-399090400d1cf689bb643956cab7f3969f7d10e7c254e5345047d02a72e2bf5f WatchSource:0}: Error finding container 399090400d1cf689bb643956cab7f3969f7d10e7c254e5345047d02a72e2bf5f: Status 404 returned error can't find the container with id 399090400d1cf689bb643956cab7f3969f7d10e7c254e5345047d02a72e2bf5f
Oct 30 00:15:01.572057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.571670    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9" event={"ID":"64908684-4f66-47fa-b0a8-bc52b38ce34d","Type":"ContainerStarted","Data":"e789207bb868713475f167d420214019d3467bd8b24d4bf320f172126dadd818"}
Oct 30 00:15:01.591735 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.591687    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" event={"ID":"e93d939d-d744-4a1b-a05c-744e65d507cb","Type":"ContainerStarted","Data":"2747861dfb5d59b5b0a4d4762f4d0b43e3efe8500b2327e4bfed51c1ba01785e"}
Oct 30 00:15:01.594018 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.593957    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8" event={"ID":"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a","Type":"ContainerStarted","Data":"73a38905b2c71ac41b0944359ce6cbc142854fed0b2fefea2e76a260325696e5"}
Oct 30 00:15:01.596454 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.595654    4441 generic.go:356] "Generic (PLEG): container finished" podID="37484057-61d0-44a9-96db-4d814905649b" containerID="e5666b8df36f3ed97ae9fb8ff8d22b413f166b52c0c196091738be96bfee841b" exitCode=0
Oct 30 00:15:01.596454 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.595723    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" event={"ID":"37484057-61d0-44a9-96db-4d814905649b","Type":"ContainerDied","Data":"e5666b8df36f3ed97ae9fb8ff8d22b413f166b52c0c196091738be96bfee841b"}
Oct 30 00:15:01.596454 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.596067    4441 scope.go:122] "RemoveContainer" containerID="e5666b8df36f3ed97ae9fb8ff8d22b413f166b52c0c196091738be96bfee841b"
Oct 30 00:15:01.597947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.597555    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:01.597947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.597622    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:01.597947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.597660    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:01.598230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.598206    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d" event={"ID":"de3547ac-b71e-4dd3-9ccc-ece14a199576","Type":"ContainerStarted","Data":"8d49c34edd3cf3e9e5a0de4aa53d08bda829970fc9d49f917c72cd2f7a2db344"}
Oct 30 00:15:01.599889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.599835    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz" event={"ID":"92f451d6-9227-4ed4-9301-3a2565e4e479","Type":"ContainerStarted","Data":"399090400d1cf689bb643956cab7f3969f7d10e7c254e5345047d02a72e2bf5f"}
Oct 30 00:15:01.763049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.750357    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:15:01.763443 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.763398    4441 patch_prober.go:29] interesting pod/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 00:15:01.763673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.763627    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 00:15:01.991114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:01.977519    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:15:02.094255 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:02.094215    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"]
Oct 30 00:15:02.617613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:02.617581    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" event={"ID":"37484057-61d0-44a9-96db-4d814905649b","Type":"ContainerStarted","Data":"2c48f73955d901fd441696d5336c495b257627ddbc324b18e7a442b22c7c8053"}
Oct 30 00:15:02.673067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:02.672946    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:15:02.673243 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:02.673097    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 00:15:02.673243 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:02.673140    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 00:15:02.744050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:02.735672    4441 patch_prober.go:29] interesting pod/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 00:15:02.744050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:02.735751    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 00:15:03.654049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:03.653428    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 00:15:03.654049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:03.653506    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 00:15:03.748065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:03.737805    4441 patch_prober.go:29] interesting pod/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 00:15:03.748065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:03.737863    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 00:15:03.866982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:03.866938    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"]
Oct 30 00:15:04.505060 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:04.482157    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:15:04.575052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:04.570290    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:15:04.661045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:04.646971    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 00:15:04.661045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:04.647043    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 00:15:04.730903 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:04.730487    4441 patch_prober.go:29] interesting pod/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 00:15:04.730903 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:04.730542    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 00:15:05.182234 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:05.162915    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"]
Oct 30 00:15:05.183139 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:05.183089    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerName="istio-proxy" containerID="cri-o://7082d1426c878647ea9055efabd6bfbb11d271f9cb878a4c927a3243b6ba92c4" gracePeriod=30
Oct 30 00:15:05.642665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:05.641719    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 00:15:05.642665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:05.641777    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 00:15:05.763208 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:05.763154    4441 patch_prober.go:29] interesting pod/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 00:15:05.763747 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:05.763224    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 00:15:06.671116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:06.670145    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 00:15:06.671116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:06.670203    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 00:15:06.733853 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:06.733399    4441 patch_prober.go:29] interesting pod/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 00:15:06.733853 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:06.733462    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 00:15:07.096057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.095404    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs" podUID="fa2ccfdf-6f75-408d-bc8e-afe656640240" containerName="payload-pre-processing" containerID="cri-o://8842e283d6d0b55a74e07209af8c475eabff326a88b79b56aca14d5c771db48f" gracePeriod=25
Oct 30 00:15:07.632771 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.631213    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 00:15:07.632771 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.631270    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 00:15:07.667669 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.667210    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"
Oct 30 00:15:07.728050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.720069    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2ccfdf-6f75-408d-bc8e-afe656640240-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2ccfdf-6f75-408d-bc8e-afe656640240-plugins-config-volume\") pod \"fa2ccfdf-6f75-408d-bc8e-afe656640240\" (UID: \"fa2ccfdf-6f75-408d-bc8e-afe656640240\") "
Oct 30 00:15:07.728050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.720120    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2ccfdf-6f75-408d-bc8e-afe656640240-kube-api-access-j9k8z\" (UniqueName: \"kubernetes.io/projected/fa2ccfdf-6f75-408d-bc8e-afe656640240-kube-api-access-j9k8z\") pod \"fa2ccfdf-6f75-408d-bc8e-afe656640240\" (UID: \"fa2ccfdf-6f75-408d-bc8e-afe656640240\") "
Oct 30 00:15:07.747072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.736665    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2ccfdf-6f75-408d-bc8e-afe656640240-plugins-config-volume" pod "fa2ccfdf-6f75-408d-bc8e-afe656640240" (UID: "fa2ccfdf-6f75-408d-bc8e-afe656640240"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:15:07.760588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.759558    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:15:07.761729 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.761675    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:15:07.768291 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.768241    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2ccfdf-6f75-408d-bc8e-afe656640240-kube-api-access-j9k8z" pod "fa2ccfdf-6f75-408d-bc8e-afe656640240" (UID: "fa2ccfdf-6f75-408d-bc8e-afe656640240"). InnerVolumeSpecName "kube-api-access-j9k8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:07.820439 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.820407    4441 generic.go:356] "Generic (PLEG): container finished" podID="fa2ccfdf-6f75-408d-bc8e-afe656640240" containerID="8842e283d6d0b55a74e07209af8c475eabff326a88b79b56aca14d5c771db48f" exitCode=0
Oct 30 00:15:07.820584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.820499    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"
Oct 30 00:15:07.820584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.820550    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs" event={"ID":"fa2ccfdf-6f75-408d-bc8e-afe656640240","Type":"ContainerDied","Data":"8842e283d6d0b55a74e07209af8c475eabff326a88b79b56aca14d5c771db48f"}
Oct 30 00:15:07.820705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.820612    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs" event={"ID":"fa2ccfdf-6f75-408d-bc8e-afe656640240","Type":"ContainerDied","Data":"89004853095e8ce3b8a0f1ab31a1ca6529ebe02e47c0282cec5a931063765deb"}
Oct 30 00:15:07.820705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.820642    4441 scope.go:122] "RemoveContainer" containerID="8842e283d6d0b55a74e07209af8c475eabff326a88b79b56aca14d5c771db48f"
Oct 30 00:15:07.825057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.821357    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2ccfdf-6f75-408d-bc8e-afe656640240-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:07.825057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.821383    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9k8z\" (UniqueName: \"kubernetes.io/projected/fa2ccfdf-6f75-408d-bc8e-afe656640240-kube-api-access-j9k8z\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:07.885206 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.885180    4441 scope.go:122] "RemoveContainer" containerID="8842e283d6d0b55a74e07209af8c475eabff326a88b79b56aca14d5c771db48f"
Oct 30 00:15:07.886219 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:07.886145    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8842e283d6d0b55a74e07209af8c475eabff326a88b79b56aca14d5c771db48f\": container with ID starting with 8842e283d6d0b55a74e07209af8c475eabff326a88b79b56aca14d5c771db48f not found: ID does not exist" containerID="8842e283d6d0b55a74e07209af8c475eabff326a88b79b56aca14d5c771db48f"
Oct 30 00:15:07.886219 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.886185    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8842e283d6d0b55a74e07209af8c475eabff326a88b79b56aca14d5c771db48f"} err="failed to get container status \"8842e283d6d0b55a74e07209af8c475eabff326a88b79b56aca14d5c771db48f\": rpc error: code = NotFound desc = could not find container \"8842e283d6d0b55a74e07209af8c475eabff326a88b79b56aca14d5c771db48f\": container with ID starting with 8842e283d6d0b55a74e07209af8c475eabff326a88b79b56aca14d5c771db48f not found: ID does not exist"
Oct 30 00:15:07.906788 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.906759    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"]
Oct 30 00:15:07.923136 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:07.923013    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-deda90ae-6988b696c9-5kbcs"]
Oct 30 00:15:08.285514 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:08.285452    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"]
Oct 30 00:15:08.286005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:08.285835    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"]
Oct 30 00:15:08.626974 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:08.626507    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2ccfdf-6f75-408d-bc8e-afe656640240" path="/var/lib/kubelet/pods/fa2ccfdf-6f75-408d-bc8e-afe656640240/volumes"
Oct 30 00:15:08.628172 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:08.627963    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 00:15:08.628917 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:08.628013    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 00:15:08.867791 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:08.867738    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq" podUID="c9e4be2a-91f9-48c9-9d2a-630a6201811b" containerName="payload-pre-processing" containerID="cri-o://37482fb1f175584e5fec90265ab575e5e761bb205ded4b47eec59dbb0b6f15ae" gracePeriod=25
Oct 30 00:15:09.257413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.257184    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"
Oct 30 00:15:09.266324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.265536    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e4be2a-91f9-48c9-9d2a-630a6201811b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e4be2a-91f9-48c9-9d2a-630a6201811b-plugins-config-volume\") pod \"c9e4be2a-91f9-48c9-9d2a-630a6201811b\" (UID: \"c9e4be2a-91f9-48c9-9d2a-630a6201811b\") "
Oct 30 00:15:09.266324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.265605    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e4be2a-91f9-48c9-9d2a-630a6201811b-kube-api-access-vwg92\" (UniqueName: \"kubernetes.io/projected/c9e4be2a-91f9-48c9-9d2a-630a6201811b-kube-api-access-vwg92\") pod \"c9e4be2a-91f9-48c9-9d2a-630a6201811b\" (UID: \"c9e4be2a-91f9-48c9-9d2a-630a6201811b\") "
Oct 30 00:15:09.266592 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.266563    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e4be2a-91f9-48c9-9d2a-630a6201811b-plugins-config-volume" pod "c9e4be2a-91f9-48c9-9d2a-630a6201811b" (UID: "c9e4be2a-91f9-48c9-9d2a-630a6201811b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:15:09.274343 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.274314    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e4be2a-91f9-48c9-9d2a-630a6201811b-kube-api-access-vwg92" pod "c9e4be2a-91f9-48c9-9d2a-630a6201811b" (UID: "c9e4be2a-91f9-48c9-9d2a-630a6201811b"). InnerVolumeSpecName "kube-api-access-vwg92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:09.367378 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.367343    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e4be2a-91f9-48c9-9d2a-630a6201811b-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:09.367378 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.367380    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwg92\" (UniqueName: \"kubernetes.io/projected/c9e4be2a-91f9-48c9-9d2a-630a6201811b-kube-api-access-vwg92\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:09.639509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.637396    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:15:09.639509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.638747    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:15:09.640276 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.639827    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:15:09.894192 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.893386    4441 generic.go:356] "Generic (PLEG): container finished" podID="c9e4be2a-91f9-48c9-9d2a-630a6201811b" containerID="37482fb1f175584e5fec90265ab575e5e761bb205ded4b47eec59dbb0b6f15ae" exitCode=0
Oct 30 00:15:09.897054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.894430    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"
Oct 30 00:15:09.897743 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.897225    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq" event={"ID":"c9e4be2a-91f9-48c9-9d2a-630a6201811b","Type":"ContainerDied","Data":"37482fb1f175584e5fec90265ab575e5e761bb205ded4b47eec59dbb0b6f15ae"}
Oct 30 00:15:09.897743 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.897294    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq" event={"ID":"c9e4be2a-91f9-48c9-9d2a-630a6201811b","Type":"ContainerDied","Data":"a6a40ea1b2e40909ea98fa96a2259f0cf043eaecdcf0be992468c989f9ef2070"}
Oct 30 00:15:09.897743 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.897328    4441 scope.go:122] "RemoveContainer" containerID="37482fb1f175584e5fec90265ab575e5e761bb205ded4b47eec59dbb0b6f15ae"
Oct 30 00:15:09.914538 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.914502    4441 scope.go:122] "RemoveContainer" containerID="37482fb1f175584e5fec90265ab575e5e761bb205ded4b47eec59dbb0b6f15ae"
Oct 30 00:15:09.915160 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:09.915132    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37482fb1f175584e5fec90265ab575e5e761bb205ded4b47eec59dbb0b6f15ae\": container with ID starting with 37482fb1f175584e5fec90265ab575e5e761bb205ded4b47eec59dbb0b6f15ae not found: ID does not exist" containerID="37482fb1f175584e5fec90265ab575e5e761bb205ded4b47eec59dbb0b6f15ae"
Oct 30 00:15:09.915278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.915171    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37482fb1f175584e5fec90265ab575e5e761bb205ded4b47eec59dbb0b6f15ae"} err="failed to get container status \"37482fb1f175584e5fec90265ab575e5e761bb205ded4b47eec59dbb0b6f15ae\": rpc error: code = NotFound desc = could not find container \"37482fb1f175584e5fec90265ab575e5e761bb205ded4b47eec59dbb0b6f15ae\": container with ID starting with 37482fb1f175584e5fec90265ab575e5e761bb205ded4b47eec59dbb0b6f15ae not found: ID does not exist"
Oct 30 00:15:09.953412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.953360    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"]
Oct 30 00:15:09.958558 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:09.958483    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-185595da-57f6764687-cmpsq"]
Oct 30 00:15:10.105975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.105936    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:15:10.547061 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.547009    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:15:10.595528 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.595496    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-token\" (UniqueName: \"kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-token\") pod \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") "
Oct 30 00:15:10.595700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.595549    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-podinfo\") pod \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") "
Oct 30 00:15:10.595700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.595590    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istiod-ca-cert\") pod \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") "
Oct 30 00:15:10.595700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.595646    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-credential-socket\") pod \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") "
Oct 30 00:15:10.595700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.595697    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-data\") pod \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") "
Oct 30 00:15:10.595920 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.595745    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-envoy\") pod \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") "
Oct 30 00:15:10.595920 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.595777    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-socket\") pod \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") "
Oct 30 00:15:10.595920 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.595864    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-certs\") pod \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") "
Oct 30 00:15:10.595920 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.595913    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-kube-api-access-7lhhm\" (UniqueName: \"kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-kube-api-access-7lhhm\") pod \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\" (UID: \"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1\") "
Oct 30 00:15:10.597176 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.596830    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-credential-socket" pod "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" (UID: "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:15:10.600706 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.600552    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-token" pod "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" (UID: "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:10.602742 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.602635    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istiod-ca-cert" pod "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" (UID: "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:15:10.603515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.603461    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-data" pod "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" (UID: "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:15:10.603760 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.603685    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-certs" pod "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" (UID: "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:15:10.604097 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.603875    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-socket" pod "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" (UID: "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:15:10.609279 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.607393    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-podinfo" pod "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" (UID: "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:15:10.609279 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.607857    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-envoy" pod "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" (UID: "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:15:10.609279 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.607897    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-kube-api-access-7lhhm" pod "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" (UID: "b4d37a88-ba16-4717-8dcd-6b6457bf4fc1"). InnerVolumeSpecName "kube-api-access-7lhhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:10.618506 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.618481    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e4be2a-91f9-48c9-9d2a-630a6201811b" path="/var/lib/kubelet/pods/c9e4be2a-91f9-48c9-9d2a-630a6201811b/volumes"
Oct 30 00:15:10.698462 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.698425    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:10.698462 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.698461    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lhhm\" (UniqueName: \"kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-kube-api-access-7lhhm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:10.698658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.698476    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:10.698658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.698491    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:10.698658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.698504    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:10.698658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.698517    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:10.698658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.698529    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:10.698658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.698543    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:10.698658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.698556    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:10.902900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.902864    4441 generic.go:356] "Generic (PLEG): container finished" podID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerID="7082d1426c878647ea9055efabd6bfbb11d271f9cb878a4c927a3243b6ba92c4" exitCode=0
Oct 30 00:15:10.903074 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.902930    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"
Oct 30 00:15:10.903074 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.902949    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" event={"ID":"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1","Type":"ContainerDied","Data":"7082d1426c878647ea9055efabd6bfbb11d271f9cb878a4c927a3243b6ba92c4"}
Oct 30 00:15:10.903074 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.903021    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt" event={"ID":"b4d37a88-ba16-4717-8dcd-6b6457bf4fc1","Type":"ContainerDied","Data":"61874f822274fe63a95e08e7412b46433769f99e559f02d552fb939501fd4f0c"}
Oct 30 00:15:10.903234 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.903076    4441 scope.go:122] "RemoveContainer" containerID="7082d1426c878647ea9055efabd6bfbb11d271f9cb878a4c927a3243b6ba92c4"
Oct 30 00:15:10.915098 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.915075    4441 scope.go:122] "RemoveContainer" containerID="38069c0470593ba688d24a5f092d27e25f18046d4aab90ee349d55cbf018f538"
Oct 30 00:15:10.935704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.935677    4441 scope.go:122] "RemoveContainer" containerID="7082d1426c878647ea9055efabd6bfbb11d271f9cb878a4c927a3243b6ba92c4"
Oct 30 00:15:10.936213 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:10.936182    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7082d1426c878647ea9055efabd6bfbb11d271f9cb878a4c927a3243b6ba92c4\": container with ID starting with 7082d1426c878647ea9055efabd6bfbb11d271f9cb878a4c927a3243b6ba92c4 not found: ID does not exist" containerID="7082d1426c878647ea9055efabd6bfbb11d271f9cb878a4c927a3243b6ba92c4"
Oct 30 00:15:10.936428 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.936397    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7082d1426c878647ea9055efabd6bfbb11d271f9cb878a4c927a3243b6ba92c4"} err="failed to get container status \"7082d1426c878647ea9055efabd6bfbb11d271f9cb878a4c927a3243b6ba92c4\": rpc error: code = NotFound desc = could not find container \"7082d1426c878647ea9055efabd6bfbb11d271f9cb878a4c927a3243b6ba92c4\": container with ID starting with 7082d1426c878647ea9055efabd6bfbb11d271f9cb878a4c927a3243b6ba92c4 not found: ID does not exist"
Oct 30 00:15:10.936556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.936533    4441 scope.go:122] "RemoveContainer" containerID="38069c0470593ba688d24a5f092d27e25f18046d4aab90ee349d55cbf018f538"
Oct 30 00:15:10.937105 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:10.936859    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38069c0470593ba688d24a5f092d27e25f18046d4aab90ee349d55cbf018f538\": container with ID starting with 38069c0470593ba688d24a5f092d27e25f18046d4aab90ee349d55cbf018f538 not found: ID does not exist" containerID="38069c0470593ba688d24a5f092d27e25f18046d4aab90ee349d55cbf018f538"
Oct 30 00:15:10.937105 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.936884    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38069c0470593ba688d24a5f092d27e25f18046d4aab90ee349d55cbf018f538"} err="failed to get container status \"38069c0470593ba688d24a5f092d27e25f18046d4aab90ee349d55cbf018f538\": rpc error: code = NotFound desc = could not find container \"38069c0470593ba688d24a5f092d27e25f18046d4aab90ee349d55cbf018f538\": container with ID starting with 38069c0470593ba688d24a5f092d27e25f18046d4aab90ee349d55cbf018f538 not found: ID does not exist"
Oct 30 00:15:10.957780 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.957751    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"]
Oct 30 00:15:10.961345 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:10.961325    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-deda90ae-openshift-default-867bfdd6d5-rgggt"]
Oct 30 00:15:11.068243 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.068164    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:15:11.068243 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.068217    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:15:11.838231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.838143    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"]
Oct 30 00:15:11.838767 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.838636    4441 cpu_manager.go:414] "containerMap: removing container" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerName="istio-proxy"
Oct 30 00:15:11.838767 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.838651    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerName="istio-proxy"
Oct 30 00:15:11.838767 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.838668    4441 cpu_manager.go:414] "containerMap: removing container" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerName="istio-proxy"
Oct 30 00:15:11.838767 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.838673    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerName="istio-proxy"
Oct 30 00:15:11.838767 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.838681    4441 cpu_manager.go:414] "containerMap: removing container" podUID="c9e4be2a-91f9-48c9-9d2a-630a6201811b" containerName="payload-pre-processing"
Oct 30 00:15:11.838767 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.838686    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e4be2a-91f9-48c9-9d2a-630a6201811b" containerName="payload-pre-processing"
Oct 30 00:15:11.838767 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.838698    4441 cpu_manager.go:414] "containerMap: removing container" podUID="fa2ccfdf-6f75-408d-bc8e-afe656640240" containerName="payload-pre-processing"
Oct 30 00:15:11.838767 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.838703    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2ccfdf-6f75-408d-bc8e-afe656640240" containerName="payload-pre-processing"
Oct 30 00:15:11.839176 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.838790    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerName="istio-proxy"
Oct 30 00:15:11.839176 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.838799    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" containerName="istio-proxy"
Oct 30 00:15:11.839176 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.838805    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e4be2a-91f9-48c9-9d2a-630a6201811b" containerName="payload-pre-processing"
Oct 30 00:15:11.839176 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.838812    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2ccfdf-6f75-408d-bc8e-afe656640240" containerName="payload-pre-processing"
Oct 30 00:15:11.841070 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.841016    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"
Oct 30 00:15:11.859529 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.859497    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-b1c718\""
Oct 30 00:15:11.860820 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.860794    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"]
Oct 30 00:15:11.909561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.909469    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8" event={"ID":"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a","Type":"ContainerStarted","Data":"e44f5520f10277224728caba09cc723fb8114d37e2e1ae5d017d0b297e10cb5a"}
Oct 30 00:15:11.911136 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.911116    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-b1c718-dockercfg-xxnbk\""
Oct 30 00:15:11.911884 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.911863    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ffa771a-b9fc-481e-8a37-9831c31780a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp\" (UID: \"7ffa771a-b9fc-481e-8a37-9831c31780a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"
Oct 30 00:15:11.911997 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.911900    4441 generic.go:356] "Generic (PLEG): container finished" podID="de3547ac-b71e-4dd3-9ccc-ece14a199576" containerID="b7efacf32daebb44b3217fe5f775b217ea1f01e9873e3911e8a2e062a5583e76" exitCode=60
Oct 30 00:15:11.911997 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.911973    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqlgr\" (UniqueName: \"kubernetes.io/projected/7ffa771a-b9fc-481e-8a37-9831c31780a2-kube-api-access-hqlgr\") pod \"payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp\" (UID: \"7ffa771a-b9fc-481e-8a37-9831c31780a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"
Oct 30 00:15:11.912152 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.911974    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d" event={"ID":"de3547ac-b71e-4dd3-9ccc-ece14a199576","Type":"ContainerDied","Data":"b7efacf32daebb44b3217fe5f775b217ea1f01e9873e3911e8a2e062a5583e76"}
Oct 30 00:15:11.912222 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.912204    4441 scope.go:122] "RemoveContainer" containerID="b7efacf32daebb44b3217fe5f775b217ea1f01e9873e3911e8a2e062a5583e76"
Oct 30 00:15:11.913478 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.913460    4441 generic.go:356] "Generic (PLEG): container finished" podID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerID="5598768fe2770195b6b21bb83e54cde9c93eb45f006f47fe6486c49f975d280d" exitCode=0
Oct 30 00:15:11.913574 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.913519    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" event={"ID":"1f550b5f-f099-42d9-bd19-66ec69fce6d5","Type":"ContainerDied","Data":"5598768fe2770195b6b21bb83e54cde9c93eb45f006f47fe6486c49f975d280d"}
Oct 30 00:15:11.913886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.913867    4441 scope.go:122] "RemoveContainer" containerID="5598768fe2770195b6b21bb83e54cde9c93eb45f006f47fe6486c49f975d280d"
Oct 30 00:15:11.915597 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.915574    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz" event={"ID":"92f451d6-9227-4ed4-9301-3a2565e4e479","Type":"ContainerStarted","Data":"bc226bcd9dbd693d2ae644f91d53f718870e74b6105293facec45c85b2952254"}
Oct 30 00:15:11.915696 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.915644    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:11.915780 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.915757    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:11.915911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.915864    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:11.940499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.939309    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9" event={"ID":"64908684-4f66-47fa-b0a8-bc52b38ce34d","Type":"ContainerStarted","Data":"8dcbfb3873ccbe3f7ff9fa9e0c865ceb9b923a9b2e3a2a53b06822fa79557665"}
Oct 30 00:15:11.954951 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.953866    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"]
Oct 30 00:15:11.964162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.963520    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"
Oct 30 00:15:11.990128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.989852    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8" podStartSLOduration=2.255031474 podStartE2EDuration="11.989829174s" totalImagesPullingTime="9.7347977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:11.984381782 +0000 UTC m=+621.509675656" watchObservedRunningTime="2026-10-30 00:15:11.989829174 +0000 UTC m=+621.515123052"
Oct 30 00:15:11.995931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:11.995894    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"]
Oct 30 00:15:12.012969 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.012935    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bca81c-b78d-45a0-8c76-13b92029347e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz\" (UID: \"21bca81c-b78d-45a0-8c76-13b92029347e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"
Oct 30 00:15:12.013282 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.013259    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb922\" (UniqueName: \"kubernetes.io/projected/21bca81c-b78d-45a0-8c76-13b92029347e-kube-api-access-nb922\") pod \"payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz\" (UID: \"21bca81c-b78d-45a0-8c76-13b92029347e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"
Oct 30 00:15:12.013435 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.013418    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ffa771a-b9fc-481e-8a37-9831c31780a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp\" (UID: \"7ffa771a-b9fc-481e-8a37-9831c31780a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"
Oct 30 00:15:12.013723 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.013704    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqlgr\" (UniqueName: \"kubernetes.io/projected/7ffa771a-b9fc-481e-8a37-9831c31780a2-kube-api-access-hqlgr\") pod \"payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp\" (UID: \"7ffa771a-b9fc-481e-8a37-9831c31780a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"
Oct 30 00:15:12.014324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.014301    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ffa771a-b9fc-481e-8a37-9831c31780a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp\" (UID: \"7ffa771a-b9fc-481e-8a37-9831c31780a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"
Oct 30 00:15:12.041651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.041585    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9" podStartSLOduration=2.3320648520000002 podStartE2EDuration="12.041565745s" totalImagesPullingTime="9.709500893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:12.040098603 +0000 UTC m=+621.565392481" watchObservedRunningTime="2026-10-30 00:15:12.041565745 +0000 UTC m=+621.566859624"
Oct 30 00:15:12.043764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.043732    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqlgr\" (UniqueName: \"kubernetes.io/projected/7ffa771a-b9fc-481e-8a37-9831c31780a2-kube-api-access-hqlgr\") pod \"payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp\" (UID: \"7ffa771a-b9fc-481e-8a37-9831c31780a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"
Oct 30 00:15:12.114512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.114482    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb922\" (UniqueName: \"kubernetes.io/projected/21bca81c-b78d-45a0-8c76-13b92029347e-kube-api-access-nb922\") pod \"payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz\" (UID: \"21bca81c-b78d-45a0-8c76-13b92029347e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"
Oct 30 00:15:12.114670 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.114653    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bca81c-b78d-45a0-8c76-13b92029347e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz\" (UID: \"21bca81c-b78d-45a0-8c76-13b92029347e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"
Oct 30 00:15:12.115505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.115474    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bca81c-b78d-45a0-8c76-13b92029347e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz\" (UID: \"21bca81c-b78d-45a0-8c76-13b92029347e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"
Oct 30 00:15:12.137756 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.137677    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb922\" (UniqueName: \"kubernetes.io/projected/21bca81c-b78d-45a0-8c76-13b92029347e-kube-api-access-nb922\") pod \"payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz\" (UID: \"21bca81c-b78d-45a0-8c76-13b92029347e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"
Oct 30 00:15:12.139716 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.139694    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"
Oct 30 00:15:12.164223 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.163931    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"
Oct 30 00:15:12.210209 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.210145    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz" podStartSLOduration=2.579518654 podStartE2EDuration="12.210124415s" totalImagesPullingTime="9.630605761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:12.207960443 +0000 UTC m=+621.733254345" watchObservedRunningTime="2026-10-30 00:15:12.210124415 +0000 UTC m=+621.735418292"
Oct 30 00:15:12.490319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.487365    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"]
Oct 30 00:15:12.490319 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:15:12.489182    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bca81c_b78d_45a0_8c76_13b92029347e.slice/crio-a8f1f238dfc49e151b1c86b0be9e3d7b8cb67bfbcb036133307d0878b8a5d68e WatchSource:0}: Error finding container a8f1f238dfc49e151b1c86b0be9e3d7b8cb67bfbcb036133307d0878b8a5d68e: Status 404 returned error can't find the container with id a8f1f238dfc49e151b1c86b0be9e3d7b8cb67bfbcb036133307d0878b8a5d68e
Oct 30 00:15:12.517947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.517919    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"]
Oct 30 00:15:12.521384 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:15:12.521352    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffa771a_b9fc_481e_8a37_9831c31780a2.slice/crio-80801fbfbdd0f4fdec841ce4f65ce7a3f63a30c51931a0ef178dcb03d826741c WatchSource:0}: Error finding container 80801fbfbdd0f4fdec841ce4f65ce7a3f63a30c51931a0ef178dcb03d826741c: Status 404 returned error can't find the container with id 80801fbfbdd0f4fdec841ce4f65ce7a3f63a30c51931a0ef178dcb03d826741c
Oct 30 00:15:12.611917 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.611884    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d37a88-ba16-4717-8dcd-6b6457bf4fc1" path="/var/lib/kubelet/pods/b4d37a88-ba16-4717-8dcd-6b6457bf4fc1/volumes"
Oct 30 00:15:12.952816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.952779    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp" event={"ID":"7ffa771a-b9fc-481e-8a37-9831c31780a2","Type":"ContainerStarted","Data":"3d9167bd5a3d7dea9a28bda44846fa7d439da6b97d199c25655432f139856b07"}
Oct 30 00:15:12.953288 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.952847    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"
Oct 30 00:15:12.953288 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.952867    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp" event={"ID":"7ffa771a-b9fc-481e-8a37-9831c31780a2","Type":"ContainerStarted","Data":"80801fbfbdd0f4fdec841ce4f65ce7a3f63a30c51931a0ef178dcb03d826741c"}
Oct 30 00:15:12.954474 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.954448    4441 generic.go:356] "Generic (PLEG): container finished" podID="92f451d6-9227-4ed4-9301-3a2565e4e479" containerID="bc226bcd9dbd693d2ae644f91d53f718870e74b6105293facec45c85b2952254" exitCode=60
Oct 30 00:15:12.954600 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.954525    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz" event={"ID":"92f451d6-9227-4ed4-9301-3a2565e4e479","Type":"ContainerDied","Data":"bc226bcd9dbd693d2ae644f91d53f718870e74b6105293facec45c85b2952254"}
Oct 30 00:15:12.954744 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.954727    4441 scope.go:122] "RemoveContainer" containerID="bc226bcd9dbd693d2ae644f91d53f718870e74b6105293facec45c85b2952254"
Oct 30 00:15:12.956116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.956089    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz" event={"ID":"21bca81c-b78d-45a0-8c76-13b92029347e","Type":"ContainerStarted","Data":"6996d4f1ff17ccc6899a9e762d0804c606cc548c0fc937cbca2ef2620378ef1d"}
Oct 30 00:15:12.956192 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.956134    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"
Oct 30 00:15:12.956192 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.956155    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz" event={"ID":"21bca81c-b78d-45a0-8c76-13b92029347e","Type":"ContainerStarted","Data":"a8f1f238dfc49e151b1c86b0be9e3d7b8cb67bfbcb036133307d0878b8a5d68e"}
Oct 30 00:15:12.957938 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.957912    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d" event={"ID":"de3547ac-b71e-4dd3-9ccc-ece14a199576","Type":"ContainerStarted","Data":"62c6365c48d4bd3b797df0985efccd7be7b7d4fcdf194fed124f47d7b0a345fc"}
Oct 30 00:15:12.959850 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.959831    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" event={"ID":"1f550b5f-f099-42d9-bd19-66ec69fce6d5","Type":"ContainerStarted","Data":"61d91671bdb37f5a84d6ee053c1e342c3cde7609e11eb5dee09bccc33ac95caa"}
Oct 30 00:15:12.961301 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.961279    4441 generic.go:356] "Generic (PLEG): container finished" podID="64908684-4f66-47fa-b0a8-bc52b38ce34d" containerID="8dcbfb3873ccbe3f7ff9fa9e0c865ceb9b923a9b2e3a2a53b06822fa79557665" exitCode=60
Oct 30 00:15:12.961397 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.961352    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9" event={"ID":"64908684-4f66-47fa-b0a8-bc52b38ce34d","Type":"ContainerDied","Data":"8dcbfb3873ccbe3f7ff9fa9e0c865ceb9b923a9b2e3a2a53b06822fa79557665"}
Oct 30 00:15:12.961510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.961491    4441 scope.go:122] "RemoveContainer" containerID="8dcbfb3873ccbe3f7ff9fa9e0c865ceb9b923a9b2e3a2a53b06822fa79557665"
Oct 30 00:15:12.962927 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.962906    4441 generic.go:356] "Generic (PLEG): container finished" podID="536c3f1d-9ca9-4bcf-bed5-d63c0affe07a" containerID="e44f5520f10277224728caba09cc723fb8114d37e2e1ae5d017d0b297e10cb5a" exitCode=60
Oct 30 00:15:12.962927 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.962926    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8" event={"ID":"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a","Type":"ContainerDied","Data":"e44f5520f10277224728caba09cc723fb8114d37e2e1ae5d017d0b297e10cb5a"}
Oct 30 00:15:12.963152 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.963134    4441 scope.go:122] "RemoveContainer" containerID="e44f5520f10277224728caba09cc723fb8114d37e2e1ae5d017d0b297e10cb5a"
Oct 30 00:15:12.997653 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:12.997610    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp" podStartSLOduration=1.997597731 podStartE2EDuration="1.997597731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:12.99488443 +0000 UTC m=+622.520178309" watchObservedRunningTime="2026-10-30 00:15:12.997597731 +0000 UTC m=+622.522891609"
Oct 30 00:15:13.120937 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.120124    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d" podStartSLOduration=3.687141928 podStartE2EDuration="13.12010189s" totalImagesPullingTime="9.432959962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:13.115175872 +0000 UTC m=+622.640469752" watchObservedRunningTime="2026-10-30 00:15:13.12010189 +0000 UTC m=+622.645395768"
Oct 30 00:15:13.159796 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.156567    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz" podStartSLOduration=2.156545883 podStartE2EDuration="2.156545883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:13.155869154 +0000 UTC m=+622.681163032" watchObservedRunningTime="2026-10-30 00:15:13.156545883 +0000 UTC m=+622.681839761"
Oct 30 00:15:13.246437 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.246349    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:15:13.247087 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.247061    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:15:13.247287 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.247262    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 00:15:13.286303 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.286271    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr" podUID="db85767a-29f1-4c0b-a912-3e4c1af1cd83" containerName="payload-pre-processing" containerID="cri-o://635ed80b1b8b4240327f641b70b7aa863af8417395d32bf47985b118d7e8c1de" gracePeriod=25
Oct 30 00:15:13.287684 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.287657    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd" podUID="95e5b2df-6caa-4421-9a87-492979195236" containerName="payload-processing" containerID="cri-o://5f507c881e2bb648419df8b3334c6ade0dd861e1960beda4a103522961dac69a" gracePeriod=25
Oct 30 00:15:13.681205 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.678231    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"
Oct 30 00:15:13.686983 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.686324    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"
Oct 30 00:15:13.765570 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.765530    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e5b2df-6caa-4421-9a87-492979195236-kube-api-access-xm269\" (UniqueName: \"kubernetes.io/projected/95e5b2df-6caa-4421-9a87-492979195236-kube-api-access-xm269\") pod \"95e5b2df-6caa-4421-9a87-492979195236\" (UID: \"95e5b2df-6caa-4421-9a87-492979195236\") "
Oct 30 00:15:13.766930 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.766764    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e5b2df-6caa-4421-9a87-492979195236-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e5b2df-6caa-4421-9a87-492979195236-plugins-config-volume\") pod \"95e5b2df-6caa-4421-9a87-492979195236\" (UID: \"95e5b2df-6caa-4421-9a87-492979195236\") "
Oct 30 00:15:13.767190 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.767166    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db85767a-29f1-4c0b-a912-3e4c1af1cd83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db85767a-29f1-4c0b-a912-3e4c1af1cd83-plugins-config-volume\") pod \"db85767a-29f1-4c0b-a912-3e4c1af1cd83\" (UID: \"db85767a-29f1-4c0b-a912-3e4c1af1cd83\") "
Oct 30 00:15:13.767408 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.767380    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e5b2df-6caa-4421-9a87-492979195236-plugins-config-volume" pod "95e5b2df-6caa-4421-9a87-492979195236" (UID: "95e5b2df-6caa-4421-9a87-492979195236"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:15:13.767548 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.767400    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db85767a-29f1-4c0b-a912-3e4c1af1cd83-kube-api-access-nj8bg\" (UniqueName: \"kubernetes.io/projected/db85767a-29f1-4c0b-a912-3e4c1af1cd83-kube-api-access-nj8bg\") pod \"db85767a-29f1-4c0b-a912-3e4c1af1cd83\" (UID: \"db85767a-29f1-4c0b-a912-3e4c1af1cd83\") "
Oct 30 00:15:13.768139 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.768117    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e5b2df-6caa-4421-9a87-492979195236-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:13.768782 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.768761    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db85767a-29f1-4c0b-a912-3e4c1af1cd83-plugins-config-volume" pod "db85767a-29f1-4c0b-a912-3e4c1af1cd83" (UID: "db85767a-29f1-4c0b-a912-3e4c1af1cd83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:15:13.770411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.770384    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db85767a-29f1-4c0b-a912-3e4c1af1cd83-kube-api-access-nj8bg" pod "db85767a-29f1-4c0b-a912-3e4c1af1cd83" (UID: "db85767a-29f1-4c0b-a912-3e4c1af1cd83"). InnerVolumeSpecName "kube-api-access-nj8bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:13.772516 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.772492    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e5b2df-6caa-4421-9a87-492979195236-kube-api-access-xm269" pod "95e5b2df-6caa-4421-9a87-492979195236" (UID: "95e5b2df-6caa-4421-9a87-492979195236"). InnerVolumeSpecName "kube-api-access-xm269". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:13.856653 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.856041    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"]
Oct 30 00:15:13.869355 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.869325    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db85767a-29f1-4c0b-a912-3e4c1af1cd83-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:13.869355 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.869353    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj8bg\" (UniqueName: \"kubernetes.io/projected/db85767a-29f1-4c0b-a912-3e4c1af1cd83-kube-api-access-nj8bg\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:13.869564 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.869365    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm269\" (UniqueName: \"kubernetes.io/projected/95e5b2df-6caa-4421-9a87-492979195236-kube-api-access-xm269\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:13.975718 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.974437    4441 generic.go:356] "Generic (PLEG): container finished" podID="de3547ac-b71e-4dd3-9ccc-ece14a199576" containerID="62c6365c48d4bd3b797df0985efccd7be7b7d4fcdf194fed124f47d7b0a345fc" exitCode=60
Oct 30 00:15:13.975718 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.974525    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d" event={"ID":"de3547ac-b71e-4dd3-9ccc-ece14a199576","Type":"ContainerDied","Data":"62c6365c48d4bd3b797df0985efccd7be7b7d4fcdf194fed124f47d7b0a345fc"}
Oct 30 00:15:13.975718 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.974572    4441 scope.go:122] "RemoveContainer" containerID="b7efacf32daebb44b3217fe5f775b217ea1f01e9873e3911e8a2e062a5583e76"
Oct 30 00:15:13.975718 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.975017    4441 scope.go:122] "RemoveContainer" containerID="62c6365c48d4bd3b797df0985efccd7be7b7d4fcdf194fed124f47d7b0a345fc"
Oct 30 00:15:13.975718 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:13.975331    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d_odh-ai-gateway-infra(de3547ac-b71e-4dd3-9ccc-ece14a199576)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d" podUID="de3547ac-b71e-4dd3-9ccc-ece14a199576"
Oct 30 00:15:13.978893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.977443    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9" event={"ID":"64908684-4f66-47fa-b0a8-bc52b38ce34d","Type":"ContainerStarted","Data":"3086f5b9c1f2d6b50e4fab4996568b12d5037bf4a6cec8e97a08fca82f028105"}
Oct 30 00:15:13.981861 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.981241    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8" event={"ID":"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a","Type":"ContainerStarted","Data":"44984c15ce0c78a08df0b12521815811d891b93d860eb82116aeec20a91d8876"}
Oct 30 00:15:13.983071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.982758    4441 generic.go:356] "Generic (PLEG): container finished" podID="95e5b2df-6caa-4421-9a87-492979195236" containerID="5f507c881e2bb648419df8b3334c6ade0dd861e1960beda4a103522961dac69a" exitCode=0
Oct 30 00:15:13.983071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.982806    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"
Oct 30 00:15:13.983071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.982825    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd" event={"ID":"95e5b2df-6caa-4421-9a87-492979195236","Type":"ContainerDied","Data":"5f507c881e2bb648419df8b3334c6ade0dd861e1960beda4a103522961dac69a"}
Oct 30 00:15:13.983071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.982864    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd" event={"ID":"95e5b2df-6caa-4421-9a87-492979195236","Type":"ContainerDied","Data":"f3f93672460889415fbb70c1ffe3a35269d68faaefe1739f38a0ea4fb796f632"}
Oct 30 00:15:13.984965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.984926    4441 generic.go:356] "Generic (PLEG): container finished" podID="92f451d6-9227-4ed4-9301-3a2565e4e479" containerID="6743082107dd227cf0e3edaea7b333cc0c888b61e12f3fc270f0789d7e540572" exitCode=60
Oct 30 00:15:13.985095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.985063    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz" event={"ID":"92f451d6-9227-4ed4-9301-3a2565e4e479","Type":"ContainerDied","Data":"6743082107dd227cf0e3edaea7b333cc0c888b61e12f3fc270f0789d7e540572"}
Oct 30 00:15:13.985633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.985404    4441 scope.go:122] "RemoveContainer" containerID="6743082107dd227cf0e3edaea7b333cc0c888b61e12f3fc270f0789d7e540572"
Oct 30 00:15:13.985633 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:13.985595    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888655-bxdjz_odh-ai-gateway-infra(92f451d6-9227-4ed4-9301-3a2565e4e479)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz" podUID="92f451d6-9227-4ed4-9301-3a2565e4e479"
Oct 30 00:15:13.986996 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.986975    4441 generic.go:356] "Generic (PLEG): container finished" podID="db85767a-29f1-4c0b-a912-3e4c1af1cd83" containerID="635ed80b1b8b4240327f641b70b7aa863af8417395d32bf47985b118d7e8c1de" exitCode=0
Oct 30 00:15:13.987239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.987214    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr" event={"ID":"db85767a-29f1-4c0b-a912-3e4c1af1cd83","Type":"ContainerDied","Data":"635ed80b1b8b4240327f641b70b7aa863af8417395d32bf47985b118d7e8c1de"}
Oct 30 00:15:13.987310 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.987282    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr" event={"ID":"db85767a-29f1-4c0b-a912-3e4c1af1cd83","Type":"ContainerDied","Data":"1613d7026dd0df794c7a0396cb452ab21f7d3d40fb38eeaad846a94e640c3a6f"}
Oct 30 00:15:13.987376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.987363    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"
Oct 30 00:15:13.992631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:13.992615    4441 scope.go:122] "RemoveContainer" containerID="5f507c881e2bb648419df8b3334c6ade0dd861e1960beda4a103522961dac69a"
Oct 30 00:15:14.000907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.000880    4441 scope.go:122] "RemoveContainer" containerID="5f507c881e2bb648419df8b3334c6ade0dd861e1960beda4a103522961dac69a"
Oct 30 00:15:14.001323 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:14.001275    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f507c881e2bb648419df8b3334c6ade0dd861e1960beda4a103522961dac69a\": container with ID starting with 5f507c881e2bb648419df8b3334c6ade0dd861e1960beda4a103522961dac69a not found: ID does not exist" containerID="5f507c881e2bb648419df8b3334c6ade0dd861e1960beda4a103522961dac69a"
Oct 30 00:15:14.001387 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.001335    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f507c881e2bb648419df8b3334c6ade0dd861e1960beda4a103522961dac69a"} err="failed to get container status \"5f507c881e2bb648419df8b3334c6ade0dd861e1960beda4a103522961dac69a\": rpc error: code = NotFound desc = could not find container \"5f507c881e2bb648419df8b3334c6ade0dd861e1960beda4a103522961dac69a\": container with ID starting with 5f507c881e2bb648419df8b3334c6ade0dd861e1960beda4a103522961dac69a not found: ID does not exist"
Oct 30 00:15:14.001387 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.001354    4441 scope.go:122] "RemoveContainer" containerID="bc226bcd9dbd693d2ae644f91d53f718870e74b6105293facec45c85b2952254"
Oct 30 00:15:14.013526 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.013501    4441 scope.go:122] "RemoveContainer" containerID="635ed80b1b8b4240327f641b70b7aa863af8417395d32bf47985b118d7e8c1de"
Oct 30 00:15:14.065477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.058673    4441 scope.go:122] "RemoveContainer" containerID="635ed80b1b8b4240327f641b70b7aa863af8417395d32bf47985b118d7e8c1de"
Oct 30 00:15:14.065477 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:14.064926    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635ed80b1b8b4240327f641b70b7aa863af8417395d32bf47985b118d7e8c1de\": container with ID starting with 635ed80b1b8b4240327f641b70b7aa863af8417395d32bf47985b118d7e8c1de not found: ID does not exist" containerID="635ed80b1b8b4240327f641b70b7aa863af8417395d32bf47985b118d7e8c1de"
Oct 30 00:15:14.065477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.064966    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635ed80b1b8b4240327f641b70b7aa863af8417395d32bf47985b118d7e8c1de"} err="failed to get container status \"635ed80b1b8b4240327f641b70b7aa863af8417395d32bf47985b118d7e8c1de\": rpc error: code = NotFound desc = could not find container \"635ed80b1b8b4240327f641b70b7aa863af8417395d32bf47985b118d7e8c1de\": container with ID starting with 635ed80b1b8b4240327f641b70b7aa863af8417395d32bf47985b118d7e8c1de not found: ID does not exist"
Oct 30 00:15:14.165610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.165515    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"]
Oct 30 00:15:14.170597 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.170568    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6c9e9fa7-5ddbfc44ff-rfclr"]
Oct 30 00:15:14.202230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.202199    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"]
Oct 30 00:15:14.204292 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.204266    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6c9e9fa7-5dd6fb758b-952cd"]
Oct 30 00:15:14.246444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.246404    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:15:14.246631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.246474    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 00:15:14.609826 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.609783    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e5b2df-6caa-4421-9a87-492979195236" path="/var/lib/kubelet/pods/95e5b2df-6caa-4421-9a87-492979195236/volumes"
Oct 30 00:15:14.610468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.610437    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db85767a-29f1-4c0b-a912-3e4c1af1cd83" path="/var/lib/kubelet/pods/db85767a-29f1-4c0b-a912-3e4c1af1cd83/volumes"
Oct 30 00:15:14.999619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:14.999585    4441 scope.go:122] "RemoveContainer" containerID="6743082107dd227cf0e3edaea7b333cc0c888b61e12f3fc270f0789d7e540572"
Oct 30 00:15:15.000590 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:15.000560    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888655-bxdjz_odh-ai-gateway-infra(92f451d6-9227-4ed4-9301-3a2565e4e479)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz" podUID="92f451d6-9227-4ed4-9301-3a2565e4e479"
Oct 30 00:15:15.007619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:15.007590    4441 scope.go:122] "RemoveContainer" containerID="62c6365c48d4bd3b797df0985efccd7be7b7d4fcdf194fed124f47d7b0a345fc"
Oct 30 00:15:15.008081 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:15.008046    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d_odh-ai-gateway-infra(de3547ac-b71e-4dd3-9ccc-ece14a199576)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d" podUID="de3547ac-b71e-4dd3-9ccc-ece14a199576"
Oct 30 00:15:15.009929 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:15.009900    4441 generic.go:356] "Generic (PLEG): container finished" podID="64908684-4f66-47fa-b0a8-bc52b38ce34d" containerID="3086f5b9c1f2d6b50e4fab4996568b12d5037bf4a6cec8e97a08fca82f028105" exitCode=60
Oct 30 00:15:15.010094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:15.010016    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9" event={"ID":"64908684-4f66-47fa-b0a8-bc52b38ce34d","Type":"ContainerDied","Data":"3086f5b9c1f2d6b50e4fab4996568b12d5037bf4a6cec8e97a08fca82f028105"}
Oct 30 00:15:15.010160 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:15.010104    4441 scope.go:122] "RemoveContainer" containerID="8dcbfb3873ccbe3f7ff9fa9e0c865ceb9b923a9b2e3a2a53b06822fa79557665"
Oct 30 00:15:15.010411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:15.010350    4441 scope.go:122] "RemoveContainer" containerID="3086f5b9c1f2d6b50e4fab4996568b12d5037bf4a6cec8e97a08fca82f028105"
Oct 30 00:15:15.010657 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:15.010622    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9_odh-ai-gateway-infra(64908684-4f66-47fa-b0a8-bc52b38ce34d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9" podUID="64908684-4f66-47fa-b0a8-bc52b38ce34d"
Oct 30 00:15:15.012859 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:15.012837    4441 generic.go:356] "Generic (PLEG): container finished" podID="536c3f1d-9ca9-4bcf-bed5-d63c0affe07a" containerID="44984c15ce0c78a08df0b12521815811d891b93d860eb82116aeec20a91d8876" exitCode=60
Oct 30 00:15:15.012983 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:15.012872    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8" event={"ID":"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a","Type":"ContainerDied","Data":"44984c15ce0c78a08df0b12521815811d891b93d860eb82116aeec20a91d8876"}
Oct 30 00:15:15.013213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:15.013195    4441 scope.go:122] "RemoveContainer" containerID="44984c15ce0c78a08df0b12521815811d891b93d860eb82116aeec20a91d8876"
Oct 30 00:15:15.013460 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:15.013431    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8_odh-ai-gateway-infra(536c3f1d-9ca9-4bcf-bed5-d63c0affe07a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8" podUID="536c3f1d-9ca9-4bcf-bed5-d63c0affe07a"
Oct 30 00:15:15.042246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:15.041877    4441 scope.go:122] "RemoveContainer" containerID="e44f5520f10277224728caba09cc723fb8114d37e2e1ae5d017d0b297e10cb5a"
Oct 30 00:15:15.246539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:15.246499    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:15:15.246799 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:15.246765    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 00:15:16.051700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:16.051670    4441 scope.go:122] "RemoveContainer" containerID="3086f5b9c1f2d6b50e4fab4996568b12d5037bf4a6cec8e97a08fca82f028105"
Oct 30 00:15:16.052171 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:16.051961    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9_odh-ai-gateway-infra(64908684-4f66-47fa-b0a8-bc52b38ce34d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9" podUID="64908684-4f66-47fa-b0a8-bc52b38ce34d"
Oct 30 00:15:16.053308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:16.053292    4441 scope.go:122] "RemoveContainer" containerID="44984c15ce0c78a08df0b12521815811d891b93d860eb82116aeec20a91d8876"
Oct 30 00:15:16.053532 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:16.053514    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8_odh-ai-gateway-infra(536c3f1d-9ca9-4bcf-bed5-d63c0affe07a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8" podUID="536c3f1d-9ca9-4bcf-bed5-d63c0affe07a"
Oct 30 00:15:16.247568 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:16.247524    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:15:16.247762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:16.247590    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 00:15:16.972175 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:16.972137    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"]
Oct 30 00:15:17.258577 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:17.258531    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:15:17.259195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:17.259152    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 00:15:17.287507 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:17.287482    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"
Oct 30 00:15:17.305417 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:17.305384    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64908684-4f66-47fa-b0a8-bc52b38ce34d-kube-api-access-rqmzc\" (UniqueName: \"kubernetes.io/projected/64908684-4f66-47fa-b0a8-bc52b38ce34d-kube-api-access-rqmzc\") pod \"64908684-4f66-47fa-b0a8-bc52b38ce34d\" (UID: \"64908684-4f66-47fa-b0a8-bc52b38ce34d\") "
Oct 30 00:15:17.305523 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:17.305437    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64908684-4f66-47fa-b0a8-bc52b38ce34d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64908684-4f66-47fa-b0a8-bc52b38ce34d-maas-api-service-ca\") pod \"64908684-4f66-47fa-b0a8-bc52b38ce34d\" (UID: \"64908684-4f66-47fa-b0a8-bc52b38ce34d\") "
Oct 30 00:15:17.305795 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:17.305774    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64908684-4f66-47fa-b0a8-bc52b38ce34d-maas-api-service-ca" pod "64908684-4f66-47fa-b0a8-bc52b38ce34d" (UID: "64908684-4f66-47fa-b0a8-bc52b38ce34d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:15:17.307595 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:17.307571    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64908684-4f66-47fa-b0a8-bc52b38ce34d-kube-api-access-rqmzc" pod "64908684-4f66-47fa-b0a8-bc52b38ce34d" (UID: "64908684-4f66-47fa-b0a8-bc52b38ce34d"). InnerVolumeSpecName "kube-api-access-rqmzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:17.407114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:17.407077    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64908684-4f66-47fa-b0a8-bc52b38ce34d-maas-api-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:17.407114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:17.407110    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqmzc\" (UniqueName: \"kubernetes.io/projected/64908684-4f66-47fa-b0a8-bc52b38ce34d-kube-api-access-rqmzc\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:18.068861 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.068827    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"
Oct 30 00:15:18.068861 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.068849    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9" event={"ID":"64908684-4f66-47fa-b0a8-bc52b38ce34d","Type":"ContainerDied","Data":"e789207bb868713475f167d420214019d3467bd8b24d4bf320f172126dadd818"}
Oct 30 00:15:18.069136 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.068916    4441 scope.go:122] "RemoveContainer" containerID="3086f5b9c1f2d6b50e4fab4996568b12d5037bf4a6cec8e97a08fca82f028105"
Oct 30 00:15:18.117605 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.117573    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"]
Oct 30 00:15:18.136784 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.136743    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-6c9e9fa7-29888655-2mcl9"]
Oct 30 00:15:18.244576 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.244534    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:15:18.244771 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.244603    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 00:15:18.380294 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.379691    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"]
Oct 30 00:15:18.380761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.380385    4441 cpu_manager.go:414] "containerMap: removing container" podUID="95e5b2df-6caa-4421-9a87-492979195236" containerName="payload-processing"
Oct 30 00:15:18.380761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.380403    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e5b2df-6caa-4421-9a87-492979195236" containerName="payload-processing"
Oct 30 00:15:18.380761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.380415    4441 cpu_manager.go:414] "containerMap: removing container" podUID="db85767a-29f1-4c0b-a912-3e4c1af1cd83" containerName="payload-pre-processing"
Oct 30 00:15:18.380761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.380423    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db85767a-29f1-4c0b-a912-3e4c1af1cd83" containerName="payload-pre-processing"
Oct 30 00:15:18.380761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.380471    4441 cpu_manager.go:414] "containerMap: removing container" podUID="64908684-4f66-47fa-b0a8-bc52b38ce34d" containerName="cleanup"
Oct 30 00:15:18.380761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.380479    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64908684-4f66-47fa-b0a8-bc52b38ce34d" containerName="cleanup"
Oct 30 00:15:18.380761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.380492    4441 cpu_manager.go:414] "containerMap: removing container" podUID="64908684-4f66-47fa-b0a8-bc52b38ce34d" containerName="cleanup"
Oct 30 00:15:18.380761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.380500    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64908684-4f66-47fa-b0a8-bc52b38ce34d" containerName="cleanup"
Oct 30 00:15:18.380761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.380612    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="64908684-4f66-47fa-b0a8-bc52b38ce34d" containerName="cleanup"
Oct 30 00:15:18.380761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.380623    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="64908684-4f66-47fa-b0a8-bc52b38ce34d" containerName="cleanup"
Oct 30 00:15:18.380761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.380635    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e5b2df-6caa-4421-9a87-492979195236" containerName="payload-processing"
Oct 30 00:15:18.380761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.380644    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="db85767a-29f1-4c0b-a912-3e4c1af1cd83" containerName="payload-pre-processing"
Oct 30 00:15:18.384189 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.384167    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.420309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.420270    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.420586 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.420555    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.420748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.420730    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.420931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.420905    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xljcd\" (UniqueName: \"kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-kube-api-access-xljcd\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.421141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.421122    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.421331 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.421314    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.421472 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.421456    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.421608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.421586    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.421807 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.421762    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.422763 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.422739    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"]
Oct 30 00:15:18.474629 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.474595    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-02665b-openshift-default-dockercfg-c72fk\""
Oct 30 00:15:18.523454 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.523423    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.523629 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.523467    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.523629 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.523503    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xljcd\" (UniqueName: \"kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-kube-api-access-xljcd\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.523629 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.523549    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.523629 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.523597    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.523629 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.523621    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.523915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.523645    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.523915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.523888    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.524043 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.523945    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.524043 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.523984    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.524043 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.523995    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.524192 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.524020    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.524301 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.524280    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.524699 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.524676    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.526146 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.526116    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.526376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.526360    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.549700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.549671    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.550129 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.550110    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xljcd\" (UniqueName: \"kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-kube-api-access-xljcd\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kvgln\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.614581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.614087    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64908684-4f66-47fa-b0a8-bc52b38ce34d" path="/var/lib/kubelet/pods/64908684-4f66-47fa-b0a8-bc52b38ce34d/volumes"
Oct 30 00:15:18.702464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.702390    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:18.868354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:18.868287    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl" podUID="d0fa9de1-11bd-49ea-9d8d-0ce5f193a621" containerName="payload-pre-processing" containerID="cri-o://f169d6c772229a3cc2f6e0584e6e8a29eaa703b10ce35b9a082661c4c2266f4e" gracePeriod=25
Oct 30 00:15:19.037876 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.037837    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"]
Oct 30 00:15:19.042322 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:15:19.042291    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377ffe8e_cc09_4c8b_a25d_c2e486236de5.slice/crio-1c779cf1487774bed98756bc1b875a22a7016ebd4535365bbf816a25a1e7acc0 WatchSource:0}: Error finding container 1c779cf1487774bed98756bc1b875a22a7016ebd4535365bbf816a25a1e7acc0: Status 404 returned error can't find the container with id 1c779cf1487774bed98756bc1b875a22a7016ebd4535365bbf816a25a1e7acc0
Oct 30 00:15:19.049243 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.049195    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:19.049405 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.049299    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:19.049405 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.049347    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:19.082812 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.082776    4441 generic.go:356] "Generic (PLEG): container finished" podID="d0fa9de1-11bd-49ea-9d8d-0ce5f193a621" containerID="f169d6c772229a3cc2f6e0584e6e8a29eaa703b10ce35b9a082661c4c2266f4e" exitCode=0
Oct 30 00:15:19.083154 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.083125    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl" event={"ID":"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621","Type":"ContainerDied","Data":"f169d6c772229a3cc2f6e0584e6e8a29eaa703b10ce35b9a082661c4c2266f4e"}
Oct 30 00:15:19.085111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.085084    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" event={"ID":"377ffe8e-cc09-4c8b-a25d-c2e486236de5","Type":"ContainerStarted","Data":"1c779cf1487774bed98756bc1b875a22a7016ebd4535365bbf816a25a1e7acc0"}
Oct 30 00:15:19.220049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.217608    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"
Oct 30 00:15:19.244100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.243200    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-plugins-config-volume\") pod \"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621\" (UID: \"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621\") "
Oct 30 00:15:19.244100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.243386    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-kube-api-access-xdvnx\" (UniqueName: \"kubernetes.io/projected/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-kube-api-access-xdvnx\") pod \"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621\" (UID: \"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621\") "
Oct 30 00:15:19.255220 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.255181    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-kube-api-access-xdvnx" pod "d0fa9de1-11bd-49ea-9d8d-0ce5f193a621" (UID: "d0fa9de1-11bd-49ea-9d8d-0ce5f193a621"). InnerVolumeSpecName "kube-api-access-xdvnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:19.255758 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.255731    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-plugins-config-volume" pod "d0fa9de1-11bd-49ea-9d8d-0ce5f193a621" (UID: "d0fa9de1-11bd-49ea-9d8d-0ce5f193a621"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:15:19.262264 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.262232    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:15:19.263575 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.263558    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:15:19.265558 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.264489    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:15:19.352778 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.346375    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdvnx\" (UniqueName: \"kubernetes.io/projected/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-kube-api-access-xdvnx\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:19.352778 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.346407    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:19.757156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.757115    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"]
Oct 30 00:15:19.757810 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.757787    4441 cpu_manager.go:414] "containerMap: removing container" podUID="d0fa9de1-11bd-49ea-9d8d-0ce5f193a621" containerName="payload-pre-processing"
Oct 30 00:15:19.757983 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.757968    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fa9de1-11bd-49ea-9d8d-0ce5f193a621" containerName="payload-pre-processing"
Oct 30 00:15:19.758261 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.758240    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0fa9de1-11bd-49ea-9d8d-0ce5f193a621" containerName="payload-pre-processing"
Oct 30 00:15:19.760828 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.760811    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.777230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.777203    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"openshift-service-ca.crt\""
Oct 30 00:15:19.777397 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.777232    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"kube-root-ca.crt\""
Oct 30 00:15:19.780162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.780136    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"]
Oct 30 00:15:19.780350 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.780329    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"premium-w0-7505bd-kserve-self-signed-certs\""
Oct 30 00:15:19.789116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.789062    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"default-dockercfg-s9svd\""
Oct 30 00:15:19.852435 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.852379    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-home\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.852619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.852526    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce95d81-779e-4c13-ab4b-04141c618347-tls-certs\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.852619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.852553    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-model-cache\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.852619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.852577    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-tmp-dir\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.852727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.852670    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-dshm\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.852727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.852693    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjct\" (UniqueName: \"kubernetes.io/projected/3ce95d81-779e-4c13-ab4b-04141c618347-kube-api-access-8mjct\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.954385 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.954340    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-dshm\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.955128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.955102    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjct\" (UniqueName: \"kubernetes.io/projected/3ce95d81-779e-4c13-ab4b-04141c618347-kube-api-access-8mjct\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.956511 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.956485    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-home\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.957088 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.957065    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce95d81-779e-4c13-ab4b-04141c618347-tls-certs\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.957219 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.957205    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-model-cache\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.957359 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.957344    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-tmp-dir\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.958150 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.956956    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-home\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.958150 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.957696    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-model-cache\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.958150 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.957914    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-tmp-dir\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.976265 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.976197    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce95d81-779e-4c13-ab4b-04141c618347-tls-certs\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.976854 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.976817    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-dshm\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:19.984584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:19.984557    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjct\" (UniqueName: \"kubernetes.io/projected/3ce95d81-779e-4c13-ab4b-04141c618347-kube-api-access-8mjct\") pod \"premium-w0-7505bd-kserve-7858c69c8f-v6c5b\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") " pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:20.075994 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:20.075910    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:20.097664 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:20.097633    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" event={"ID":"377ffe8e-cc09-4c8b-a25d-c2e486236de5","Type":"ContainerStarted","Data":"a25a1c66f47386d6aa90a9c7ba8dd7a6bbcb697db8ec209a467a7cc59eea78f0"}
Oct 30 00:15:20.098928 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:20.098903    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"
Oct 30 00:15:20.099069 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:20.098929    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl" event={"ID":"d0fa9de1-11bd-49ea-9d8d-0ce5f193a621","Type":"ContainerDied","Data":"8b778f9e837b58f7992951fdac1341c517cf07927c9ada85825e8c86a9e7641a"}
Oct 30 00:15:20.099069 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:20.098978    4441 scope.go:122] "RemoveContainer" containerID="f169d6c772229a3cc2f6e0584e6e8a29eaa703b10ce35b9a082661c4c2266f4e"
Oct 30 00:15:20.153944 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:20.153891    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" podStartSLOduration=2.153879039 podStartE2EDuration="2.153879039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:20.15252913 +0000 UTC m=+629.677823006" watchObservedRunningTime="2026-10-30 00:15:20.153879039 +0000 UTC m=+629.679172920"
Oct 30 00:15:20.187060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:20.187013    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"]
Oct 30 00:15:20.190414 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:20.190391    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ffe924e-5cd88c8567-x7jpl"]
Oct 30 00:15:20.411947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:20.411920    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"]
Oct 30 00:15:20.414164 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:15:20.414136    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ce95d81_779e_4c13_ab4b_04141c618347.slice/crio-7e4cef068166935e2f9f4390f42a22eebc0a38b8f2cabf1c40819e1cd07e12fb WatchSource:0}: Error finding container 7e4cef068166935e2f9f4390f42a22eebc0a38b8f2cabf1c40819e1cd07e12fb: Status 404 returned error can't find the container with id 7e4cef068166935e2f9f4390f42a22eebc0a38b8f2cabf1c40819e1cd07e12fb
Oct 30 00:15:20.607935 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:20.607904    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0fa9de1-11bd-49ea-9d8d-0ce5f193a621" path="/var/lib/kubelet/pods/d0fa9de1-11bd-49ea-9d8d-0ce5f193a621/volumes"
Oct 30 00:15:20.703013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:20.702945    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:20.708250 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:20.708227    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:21.112738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:21.112702    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" event={"ID":"3ce95d81-779e-4c13-ab4b-04141c618347","Type":"ContainerStarted","Data":"22f753964d639f8c736b1a25a298ad7c14b98cc9fd771d6808272e7fa0643875"}
Oct 30 00:15:21.112738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:21.112751    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" event={"ID":"3ce95d81-779e-4c13-ab4b-04141c618347","Type":"ContainerStarted","Data":"7e4cef068166935e2f9f4390f42a22eebc0a38b8f2cabf1c40819e1cd07e12fb"}
Oct 30 00:15:21.113864 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:21.113843    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:21.114804 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:21.114784    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:21.151564 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:21.151505    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podStartSLOduration=2.1514906639999998 podStartE2EDuration="2.151490664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:21.150927888 +0000 UTC m=+630.676221777" watchObservedRunningTime="2026-10-30 00:15:21.151490664 +0000 UTC m=+630.676784537"
Oct 30 00:15:22.076282 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.076246    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:22.367819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.367727    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"]
Oct 30 00:15:22.368298 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.367938    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w" podUID="1994b6ff-5502-487b-8ca0-33d53ac8d41b" containerName="maas-api" containerID="cri-o://880e5ceda44086781d4e165e02b8afeb8def027dafc75348e9ed04a7c527e356" gracePeriod=30
Oct 30 00:15:22.436136 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.436103    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"]
Oct 30 00:15:22.598145 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.597982    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"
Oct 30 00:15:22.649628 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.649587    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:15:22.689292 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.689264    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-maas-api-service-ca\") pod \"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a\" (UID: \"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a\") "
Oct 30 00:15:22.689431 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.689355    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-metrics-certs\") pod \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") "
Oct 30 00:15:22.689431 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.689381    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1994b6ff-5502-487b-8ca0-33d53ac8d41b-kube-api-access-5q4hr\" (UniqueName: \"kubernetes.io/projected/1994b6ff-5502-487b-8ca0-33d53ac8d41b-kube-api-access-5q4hr\") pod \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") "
Oct 30 00:15:22.689431 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.689411    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-kube-api-access-rm7vg\" (UniqueName: \"kubernetes.io/projected/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-kube-api-access-rm7vg\") pod \"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a\" (UID: \"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a\") "
Oct 30 00:15:22.689551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.689447    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-maas-api-tls\") pod \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") "
Oct 30 00:15:22.689551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.689528    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1994b6ff-5502-487b-8ca0-33d53ac8d41b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1994b6ff-5502-487b-8ca0-33d53ac8d41b-openshift-service-ca\") pod \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\" (UID: \"1994b6ff-5502-487b-8ca0-33d53ac8d41b\") "
Oct 30 00:15:22.689657 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.689602    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-maas-api-service-ca" pod "536c3f1d-9ca9-4bcf-bed5-d63c0affe07a" (UID: "536c3f1d-9ca9-4bcf-bed5-d63c0affe07a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:15:22.689927 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.689906    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-maas-api-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:22.690072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.689933    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1994b6ff-5502-487b-8ca0-33d53ac8d41b-openshift-service-ca" pod "1994b6ff-5502-487b-8ca0-33d53ac8d41b" (UID: "1994b6ff-5502-487b-8ca0-33d53ac8d41b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:15:22.691648 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.691624    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-metrics-certs" pod "1994b6ff-5502-487b-8ca0-33d53ac8d41b" (UID: "1994b6ff-5502-487b-8ca0-33d53ac8d41b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:15:22.691648 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.691637    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-kube-api-access-rm7vg" pod "536c3f1d-9ca9-4bcf-bed5-d63c0affe07a" (UID: "536c3f1d-9ca9-4bcf-bed5-d63c0affe07a"). InnerVolumeSpecName "kube-api-access-rm7vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:22.691648 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.691650    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1994b6ff-5502-487b-8ca0-33d53ac8d41b-kube-api-access-5q4hr" pod "1994b6ff-5502-487b-8ca0-33d53ac8d41b" (UID: "1994b6ff-5502-487b-8ca0-33d53ac8d41b"). InnerVolumeSpecName "kube-api-access-5q4hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:22.691817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.691664    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-maas-api-tls" pod "1994b6ff-5502-487b-8ca0-33d53ac8d41b" (UID: "1994b6ff-5502-487b-8ca0-33d53ac8d41b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:15:22.790989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.790950    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:22.790989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.790976    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q4hr\" (UniqueName: \"kubernetes.io/projected/1994b6ff-5502-487b-8ca0-33d53ac8d41b-kube-api-access-5q4hr\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:22.790989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.790986    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm7vg\" (UniqueName: \"kubernetes.io/projected/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a-kube-api-access-rm7vg\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:22.790989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.790995    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1994b6ff-5502-487b-8ca0-33d53ac8d41b-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:22.790989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:22.791003    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1994b6ff-5502-487b-8ca0-33d53ac8d41b-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:23.127425 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.127392    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"
Oct 30 00:15:23.127614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.127394    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8" event={"ID":"536c3f1d-9ca9-4bcf-bed5-d63c0affe07a","Type":"ContainerDied","Data":"73a38905b2c71ac41b0944359ce6cbc142854fed0b2fefea2e76a260325696e5"}
Oct 30 00:15:23.127614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.127530    4441 scope.go:122] "RemoveContainer" containerID="44984c15ce0c78a08df0b12521815811d891b93d860eb82116aeec20a91d8876"
Oct 30 00:15:23.128866 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.128840    4441 generic.go:356] "Generic (PLEG): container finished" podID="1994b6ff-5502-487b-8ca0-33d53ac8d41b" containerID="880e5ceda44086781d4e165e02b8afeb8def027dafc75348e9ed04a7c527e356" exitCode=0
Oct 30 00:15:23.128970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.128886    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"
Oct 30 00:15:23.128970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.128929    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w" event={"ID":"1994b6ff-5502-487b-8ca0-33d53ac8d41b","Type":"ContainerDied","Data":"880e5ceda44086781d4e165e02b8afeb8def027dafc75348e9ed04a7c527e356"}
Oct 30 00:15:23.129133 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.128985    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w" event={"ID":"1994b6ff-5502-487b-8ca0-33d53ac8d41b","Type":"ContainerDied","Data":"f7b24e514f049d5fcb18a9685707f1cf4e7db8b73e4f5aff1bf1b7c0857a0cea"}
Oct 30 00:15:23.137768 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.137750    4441 scope.go:122] "RemoveContainer" containerID="880e5ceda44086781d4e165e02b8afeb8def027dafc75348e9ed04a7c527e356"
Oct 30 00:15:23.147527 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.147509    4441 scope.go:122] "RemoveContainer" containerID="880e5ceda44086781d4e165e02b8afeb8def027dafc75348e9ed04a7c527e356"
Oct 30 00:15:23.147808 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:23.147788    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880e5ceda44086781d4e165e02b8afeb8def027dafc75348e9ed04a7c527e356\": container with ID starting with 880e5ceda44086781d4e165e02b8afeb8def027dafc75348e9ed04a7c527e356 not found: ID does not exist" containerID="880e5ceda44086781d4e165e02b8afeb8def027dafc75348e9ed04a7c527e356"
Oct 30 00:15:23.147858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.147817    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880e5ceda44086781d4e165e02b8afeb8def027dafc75348e9ed04a7c527e356"} err="failed to get container status \"880e5ceda44086781d4e165e02b8afeb8def027dafc75348e9ed04a7c527e356\": rpc error: code = NotFound desc = could not find container \"880e5ceda44086781d4e165e02b8afeb8def027dafc75348e9ed04a7c527e356\": container with ID starting with 880e5ceda44086781d4e165e02b8afeb8def027dafc75348e9ed04a7c527e356 not found: ID does not exist"
Oct 30 00:15:23.180285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.180256    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"]
Oct 30 00:15:23.191327 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.191306    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ffe924e-6984c5454d-jj49w"]
Oct 30 00:15:23.223739 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.223710    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"]
Oct 30 00:15:23.229678 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.229654    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8ffe924e-29888655-pxqg8"]
Oct 30 00:15:23.330416 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.330379    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"]
Oct 30 00:15:23.330645 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.330622    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerName="istio-proxy" containerID="cri-o://2747861dfb5d59b5b0a4d4762f4d0b43e3efe8500b2327e4bfed51c1ba01785e" gracePeriod=30
Oct 30 00:15:23.991253 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.991224    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"
Oct 30 00:15:23.991798 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:23.991779    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"
Oct 30 00:15:24.411303 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.411221    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"]
Oct 30 00:15:24.411763 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.411746    4441 cpu_manager.go:414] "containerMap: removing container" podUID="1994b6ff-5502-487b-8ca0-33d53ac8d41b" containerName="maas-api"
Oct 30 00:15:24.411806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.411763    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1994b6ff-5502-487b-8ca0-33d53ac8d41b" containerName="maas-api"
Oct 30 00:15:24.411806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.411777    4441 cpu_manager.go:414] "containerMap: removing container" podUID="536c3f1d-9ca9-4bcf-bed5-d63c0affe07a" containerName="cleanup"
Oct 30 00:15:24.411806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.411785    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536c3f1d-9ca9-4bcf-bed5-d63c0affe07a" containerName="cleanup"
Oct 30 00:15:24.411806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.411793    4441 cpu_manager.go:414] "containerMap: removing container" podUID="536c3f1d-9ca9-4bcf-bed5-d63c0affe07a" containerName="cleanup"
Oct 30 00:15:24.411806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.411799    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536c3f1d-9ca9-4bcf-bed5-d63c0affe07a" containerName="cleanup"
Oct 30 00:15:24.411969 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.411875    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1994b6ff-5502-487b-8ca0-33d53ac8d41b" containerName="maas-api"
Oct 30 00:15:24.411969 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.411882    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="536c3f1d-9ca9-4bcf-bed5-d63c0affe07a" containerName="cleanup"
Oct 30 00:15:24.411969 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.411888    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="536c3f1d-9ca9-4bcf-bed5-d63c0affe07a" containerName="cleanup"
Oct 30 00:15:24.415384 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.415365    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"
Oct 30 00:15:24.429485 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.429461    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-02665b-dockercfg-wrxlq\""
Oct 30 00:15:24.429664 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.429558    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-02665b\""
Oct 30 00:15:24.432253 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.432229    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"]
Oct 30 00:15:24.510115 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.510076    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759af21c-c080-4627-8863-b3f419cd4420-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm\" (UID: \"759af21c-c080-4627-8863-b3f419cd4420\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"
Oct 30 00:15:24.510558 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.510532    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rw7p\" (UniqueName: \"kubernetes.io/projected/759af21c-c080-4627-8863-b3f419cd4420-kube-api-access-7rw7p\") pod \"payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm\" (UID: \"759af21c-c080-4627-8863-b3f419cd4420\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"
Oct 30 00:15:24.602200 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.602173    4441 scope.go:122] "RemoveContainer" containerID="62c6365c48d4bd3b797df0985efccd7be7b7d4fcdf194fed124f47d7b0a345fc"
Oct 30 00:15:24.609667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.609639    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1994b6ff-5502-487b-8ca0-33d53ac8d41b" path="/var/lib/kubelet/pods/1994b6ff-5502-487b-8ca0-33d53ac8d41b/volumes"
Oct 30 00:15:24.610227 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.610208    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536c3f1d-9ca9-4bcf-bed5-d63c0affe07a" path="/var/lib/kubelet/pods/536c3f1d-9ca9-4bcf-bed5-d63c0affe07a/volumes"
Oct 30 00:15:24.617782 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.617750    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759af21c-c080-4627-8863-b3f419cd4420-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm\" (UID: \"759af21c-c080-4627-8863-b3f419cd4420\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"
Oct 30 00:15:24.617915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.617840    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rw7p\" (UniqueName: \"kubernetes.io/projected/759af21c-c080-4627-8863-b3f419cd4420-kube-api-access-7rw7p\") pod \"payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm\" (UID: \"759af21c-c080-4627-8863-b3f419cd4420\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"
Oct 30 00:15:24.618676 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.618653    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759af21c-c080-4627-8863-b3f419cd4420-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm\" (UID: \"759af21c-c080-4627-8863-b3f419cd4420\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"
Oct 30 00:15:24.628760 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.628730    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"]
Oct 30 00:15:24.632903 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.632883    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.664231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.664166    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-564c8f-openshift-default-dockercfg-f7gl6\""
Oct 30 00:15:24.669460 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.669432    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"]
Oct 30 00:15:24.670323 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.670141    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rw7p\" (UniqueName: \"kubernetes.io/projected/759af21c-c080-4627-8863-b3f419cd4420-kube-api-access-7rw7p\") pod \"payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm\" (UID: \"759af21c-c080-4627-8863-b3f419cd4420\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"
Oct 30 00:15:24.718408 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.718374    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.718618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.718421    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daae17d4-1319-4ef7-adb7-a013358915db-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.718618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.718544    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.718618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.718581    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkch\" (UniqueName: \"kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-kube-api-access-zpkch\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.718810 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.718722    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.718860 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.718848    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.718917 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.718867    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.718917 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.718895    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.719082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.718957    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daae17d4-1319-4ef7-adb7-a013358915db-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.728270 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.728239    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"
Oct 30 00:15:24.820620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.820580    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.820620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.820619    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.820804 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.820642    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.820962 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.820921    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daae17d4-1319-4ef7-adb7-a013358915db-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.821150 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.820990    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.821150 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.821063    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daae17d4-1319-4ef7-adb7-a013358915db-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.821150 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.821128    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.821313 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.821154    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkch\" (UniqueName: \"kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-kube-api-access-zpkch\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.821313 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.821226    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.821481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.821456    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.822775 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.820932    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.822775 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.822383    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.822775 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.822415    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daae17d4-1319-4ef7-adb7-a013358915db-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.822992 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.822909    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.826013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.825919    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daae17d4-1319-4ef7-adb7-a013358915db-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.829598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.829574    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.844402 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.844382    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkch\" (UniqueName: \"kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-kube-api-access-zpkch\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.844935 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.844912    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:24.958592 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:24.958567    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:25.063677 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:25.063653    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"]
Oct 30 00:15:25.065357 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:15:25.065332    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759af21c_c080_4627_8863_b3f419cd4420.slice/crio-ce6d4495c4cf1d77bba5becf2027a8b4539709e8096552f77646f5547cdfabf8 WatchSource:0}: Error finding container ce6d4495c4cf1d77bba5becf2027a8b4539709e8096552f77646f5547cdfabf8: Status 404 returned error can't find the container with id ce6d4495c4cf1d77bba5becf2027a8b4539709e8096552f77646f5547cdfabf8
Oct 30 00:15:25.143595 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:25.143556    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d" event={"ID":"de3547ac-b71e-4dd3-9ccc-ece14a199576","Type":"ContainerStarted","Data":"8a21daa192654ab0912cfc490997aef454bcbd8273afbb87a62eab274b23538a"}
Oct 30 00:15:25.145155 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:25.145129    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm" event={"ID":"759af21c-c080-4627-8863-b3f419cd4420","Type":"ContainerStarted","Data":"8a85a64d901ba20f90974954493e7cb45e4068bbc1fdbb6786a7874562504a2d"}
Oct 30 00:15:25.145288 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:25.145173    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm" event={"ID":"759af21c-c080-4627-8863-b3f419cd4420","Type":"ContainerStarted","Data":"ce6d4495c4cf1d77bba5becf2027a8b4539709e8096552f77646f5547cdfabf8"}
Oct 30 00:15:25.145288 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:25.145207    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"
Oct 30 00:15:25.220965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:25.220856    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm" podStartSLOduration=1.2208366449999999 podStartE2EDuration="1.220836645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:25.219355839 +0000 UTC m=+634.744649717" watchObservedRunningTime="2026-10-30 00:15:25.220836645 +0000 UTC m=+634.746130528"
Oct 30 00:15:25.257880 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:25.257855    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"]
Oct 30 00:15:25.260832 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:15:25.260805    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaae17d4_1319_4ef7_adb7_a013358915db.slice/crio-4710b885863f962d4cc3e20be947299ae057e12c9122998108ac463e664e6e3d WatchSource:0}: Error finding container 4710b885863f962d4cc3e20be947299ae057e12c9122998108ac463e664e6e3d: Status 404 returned error can't find the container with id 4710b885863f962d4cc3e20be947299ae057e12c9122998108ac463e664e6e3d
Oct 30 00:15:25.263814 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:25.263784    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:25.263908 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:25.263848    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:25.263908 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:25.263878    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:26.153793 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.153736    4441 generic.go:356] "Generic (PLEG): container finished" podID="de3547ac-b71e-4dd3-9ccc-ece14a199576" containerID="8a21daa192654ab0912cfc490997aef454bcbd8273afbb87a62eab274b23538a" exitCode=60
Oct 30 00:15:26.154349 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.153896    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d" event={"ID":"de3547ac-b71e-4dd3-9ccc-ece14a199576","Type":"ContainerDied","Data":"8a21daa192654ab0912cfc490997aef454bcbd8273afbb87a62eab274b23538a"}
Oct 30 00:15:26.154349 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.153961    4441 scope.go:122] "RemoveContainer" containerID="62c6365c48d4bd3b797df0985efccd7be7b7d4fcdf194fed124f47d7b0a345fc"
Oct 30 00:15:26.154491 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.154469    4441 scope.go:122] "RemoveContainer" containerID="8a21daa192654ab0912cfc490997aef454bcbd8273afbb87a62eab274b23538a"
Oct 30 00:15:26.154954 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:26.154771    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d_odh-ai-gateway-infra(de3547ac-b71e-4dd3-9ccc-ece14a199576)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d" podUID="de3547ac-b71e-4dd3-9ccc-ece14a199576"
Oct 30 00:15:26.156494 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.156467    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" event={"ID":"daae17d4-1319-4ef7-adb7-a013358915db","Type":"ContainerStarted","Data":"9b02c56d3aad4dd1c1ec9d2b66eed5a6acf66073ad5f732b3a4452bde113c547"}
Oct 30 00:15:26.156597 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.156520    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" event={"ID":"daae17d4-1319-4ef7-adb7-a013358915db","Type":"ContainerStarted","Data":"4710b885863f962d4cc3e20be947299ae057e12c9122998108ac463e664e6e3d"}
Oct 30 00:15:26.192054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.191993    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"]
Oct 30 00:15:26.236543 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.236512    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"]
Oct 30 00:15:26.240908 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.240882    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"
Oct 30 00:15:26.250139 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.250072    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" podStartSLOduration=2.250054521 podStartE2EDuration="2.250054521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:26.246554882 +0000 UTC m=+635.771848762" watchObservedRunningTime="2026-10-30 00:15:26.250054521 +0000 UTC m=+635.775348398"
Oct 30 00:15:26.258284 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.258069    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"]
Oct 30 00:15:26.261076 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.260907    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-322d20\""
Oct 30 00:15:26.261076 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.260907    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-322d20-dockercfg-qpqzv\""
Oct 30 00:15:26.338399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.338355    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g89zt\" (UniqueName: \"kubernetes.io/projected/13e3cf7a-4a8c-476e-93ec-2a1734dad886-kube-api-access-g89zt\") pod \"payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm\" (UID: \"13e3cf7a-4a8c-476e-93ec-2a1734dad886\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"
Oct 30 00:15:26.338598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.338505    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e3cf7a-4a8c-476e-93ec-2a1734dad886-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm\" (UID: \"13e3cf7a-4a8c-476e-93ec-2a1734dad886\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"
Oct 30 00:15:26.439774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.439687    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g89zt\" (UniqueName: \"kubernetes.io/projected/13e3cf7a-4a8c-476e-93ec-2a1734dad886-kube-api-access-g89zt\") pod \"payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm\" (UID: \"13e3cf7a-4a8c-476e-93ec-2a1734dad886\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"
Oct 30 00:15:26.439938 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.439816    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e3cf7a-4a8c-476e-93ec-2a1734dad886-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm\" (UID: \"13e3cf7a-4a8c-476e-93ec-2a1734dad886\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"
Oct 30 00:15:26.440746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.440721    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e3cf7a-4a8c-476e-93ec-2a1734dad886-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm\" (UID: \"13e3cf7a-4a8c-476e-93ec-2a1734dad886\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"
Oct 30 00:15:26.462939 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.462905    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g89zt\" (UniqueName: \"kubernetes.io/projected/13e3cf7a-4a8c-476e-93ec-2a1734dad886-kube-api-access-g89zt\") pod \"payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm\" (UID: \"13e3cf7a-4a8c-476e-93ec-2a1734dad886\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"
Oct 30 00:15:26.562003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.561962    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"
Oct 30 00:15:26.649475 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.649435    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:15:26.934966 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.934935    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"]
Oct 30 00:15:26.937307 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:15:26.937268    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e3cf7a_4a8c_476e_93ec_2a1734dad886.slice/crio-1b15e83f28ec4c8a95c8b1e448cc2a265277177257131cb1b19cb7fa1f7e9de5 WatchSource:0}: Error finding container 1b15e83f28ec4c8a95c8b1e448cc2a265277177257131cb1b19cb7fa1f7e9de5: Status 404 returned error can't find the container with id 1b15e83f28ec4c8a95c8b1e448cc2a265277177257131cb1b19cb7fa1f7e9de5
Oct 30 00:15:26.960928 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:26.960883    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:27.015583 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.015557    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:27.169108 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.168984    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm" event={"ID":"13e3cf7a-4a8c-476e-93ec-2a1734dad886","Type":"ContainerStarted","Data":"cae99119b8542e3d7d3ae8527046f883dd7506151b70a7fffe22e862a48e306f"}
Oct 30 00:15:27.169108 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.169070    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm" event={"ID":"13e3cf7a-4a8c-476e-93ec-2a1734dad886","Type":"ContainerStarted","Data":"1b15e83f28ec4c8a95c8b1e448cc2a265277177257131cb1b19cb7fa1f7e9de5"}
Oct 30 00:15:27.169683 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.169257    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:27.169683 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.169304    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"
Oct 30 00:15:27.170284 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.170261    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:15:27.345890 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.345743    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm" podStartSLOduration=1.345722837 podStartE2EDuration="1.345722837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:27.344274345 +0000 UTC m=+636.869568222" watchObservedRunningTime="2026-10-30 00:15:27.345722837 +0000 UTC m=+636.871016714"
Oct 30 00:15:27.412750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.412721    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"
Oct 30 00:15:27.461969 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.461790    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de3547ac-b71e-4dd3-9ccc-ece14a199576-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de3547ac-b71e-4dd3-9ccc-ece14a199576-maas-api-service-ca\") pod \"de3547ac-b71e-4dd3-9ccc-ece14a199576\" (UID: \"de3547ac-b71e-4dd3-9ccc-ece14a199576\") "
Oct 30 00:15:27.461969 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.461905    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3547ac-b71e-4dd3-9ccc-ece14a199576-kube-api-access-qxxh8\" (UniqueName: \"kubernetes.io/projected/de3547ac-b71e-4dd3-9ccc-ece14a199576-kube-api-access-qxxh8\") pod \"de3547ac-b71e-4dd3-9ccc-ece14a199576\" (UID: \"de3547ac-b71e-4dd3-9ccc-ece14a199576\") "
Oct 30 00:15:27.468268 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.468215    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3547ac-b71e-4dd3-9ccc-ece14a199576-maas-api-service-ca" pod "de3547ac-b71e-4dd3-9ccc-ece14a199576" (UID: "de3547ac-b71e-4dd3-9ccc-ece14a199576"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:15:27.477349 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.477307    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3547ac-b71e-4dd3-9ccc-ece14a199576-kube-api-access-qxxh8" pod "de3547ac-b71e-4dd3-9ccc-ece14a199576" (UID: "de3547ac-b71e-4dd3-9ccc-ece14a199576"). InnerVolumeSpecName "kube-api-access-qxxh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:27.563898 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.563801    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de3547ac-b71e-4dd3-9ccc-ece14a199576-maas-api-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:27.563898 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:27.563831    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxxh8\" (UniqueName: \"kubernetes.io/projected/de3547ac-b71e-4dd3-9ccc-ece14a199576-kube-api-access-qxxh8\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:28.176615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.176586    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"
Oct 30 00:15:28.177135 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.176617    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d" event={"ID":"de3547ac-b71e-4dd3-9ccc-ece14a199576","Type":"ContainerDied","Data":"8d49c34edd3cf3e9e5a0de4aa53d08bda829970fc9d49f917c72cd2f7a2db344"}
Oct 30 00:15:28.177135 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.176692    4441 scope.go:122] "RemoveContainer" containerID="8a21daa192654ab0912cfc490997aef454bcbd8273afbb87a62eab274b23538a"
Oct 30 00:15:28.230083 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.230046    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"]
Oct 30 00:15:28.241362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.241329    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888655-rlj8d"]
Oct 30 00:15:28.599482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.599447    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:15:28.601946 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.601922    4441 scope.go:122] "RemoveContainer" containerID="6743082107dd227cf0e3edaea7b333cc0c888b61e12f3fc270f0789d7e540572"
Oct 30 00:15:28.608640 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.608615    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de3547ac-b71e-4dd3-9ccc-ece14a199576" path="/var/lib/kubelet/pods/de3547ac-b71e-4dd3-9ccc-ece14a199576/volumes"
Oct 30 00:15:28.682973 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.682929    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-credential-socket\") pod \"e93d939d-d744-4a1b-a05c-744e65d507cb\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") "
Oct 30 00:15:28.682973 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.682980    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-certs\") pod \"e93d939d-d744-4a1b-a05c-744e65d507cb\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") "
Oct 30 00:15:28.683308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683007    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-data\") pod \"e93d939d-d744-4a1b-a05c-744e65d507cb\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") "
Oct 30 00:15:28.683308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683059    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-podinfo\") pod \"e93d939d-d744-4a1b-a05c-744e65d507cb\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") "
Oct 30 00:15:28.683308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683096    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-token\" (UniqueName: \"kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-token\") pod \"e93d939d-d744-4a1b-a05c-744e65d507cb\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") "
Oct 30 00:15:28.683308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683169    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-kube-api-access-kc927\" (UniqueName: \"kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-kube-api-access-kc927\") pod \"e93d939d-d744-4a1b-a05c-744e65d507cb\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") "
Oct 30 00:15:28.683308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683197    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-socket\") pod \"e93d939d-d744-4a1b-a05c-744e65d507cb\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") "
Oct 30 00:15:28.683308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683257    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93d939d-d744-4a1b-a05c-744e65d507cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e93d939d-d744-4a1b-a05c-744e65d507cb-istiod-ca-cert\") pod \"e93d939d-d744-4a1b-a05c-744e65d507cb\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") "
Oct 30 00:15:28.683308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683289    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-envoy\") pod \"e93d939d-d744-4a1b-a05c-744e65d507cb\" (UID: \"e93d939d-d744-4a1b-a05c-744e65d507cb\") "
Oct 30 00:15:28.683684 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683331    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-data" pod "e93d939d-d744-4a1b-a05c-744e65d507cb" (UID: "e93d939d-d744-4a1b-a05c-744e65d507cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:15:28.683684 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683344    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-certs" pod "e93d939d-d744-4a1b-a05c-744e65d507cb" (UID: "e93d939d-d744-4a1b-a05c-744e65d507cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:15:28.683684 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683622    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-socket" pod "e93d939d-d744-4a1b-a05c-744e65d507cb" (UID: "e93d939d-d744-4a1b-a05c-744e65d507cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:15:28.683839 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683799    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:28.683839 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683822    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:28.683839 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683836    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:28.683999 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.683847    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-credential-socket" pod "e93d939d-d744-4a1b-a05c-744e65d507cb" (UID: "e93d939d-d744-4a1b-a05c-744e65d507cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:15:28.684397 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.684096    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93d939d-d744-4a1b-a05c-744e65d507cb-istiod-ca-cert" pod "e93d939d-d744-4a1b-a05c-744e65d507cb" (UID: "e93d939d-d744-4a1b-a05c-744e65d507cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:15:28.722450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.722374    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-token" pod "e93d939d-d744-4a1b-a05c-744e65d507cb" (UID: "e93d939d-d744-4a1b-a05c-744e65d507cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:28.722585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.722453    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-kube-api-access-kc927" pod "e93d939d-d744-4a1b-a05c-744e65d507cb" (UID: "e93d939d-d744-4a1b-a05c-744e65d507cb"). InnerVolumeSpecName "kube-api-access-kc927". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:28.722737 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.722719    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-envoy" pod "e93d939d-d744-4a1b-a05c-744e65d507cb" (UID: "e93d939d-d744-4a1b-a05c-744e65d507cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:15:28.722737 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.722729    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-podinfo" pod "e93d939d-d744-4a1b-a05c-744e65d507cb" (UID: "e93d939d-d744-4a1b-a05c-744e65d507cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:15:28.784568 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.784528    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:28.784568 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.784553    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:28.784568 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.784563    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc927\" (UniqueName: \"kubernetes.io/projected/e93d939d-d744-4a1b-a05c-744e65d507cb-kube-api-access-kc927\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:28.784568 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.784573    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e93d939d-d744-4a1b-a05c-744e65d507cb-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:28.784568 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.784582    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:28.784949 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:28.784590    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93d939d-d744-4a1b-a05c-744e65d507cb-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:29.188565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:29.188518    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz" event={"ID":"92f451d6-9227-4ed4-9301-3a2565e4e479","Type":"ContainerStarted","Data":"291a0ea4530c205ca6ab7e9e668f0e813ae834cb1d93d8378144e858ba8d0dc0"}
Oct 30 00:15:29.190295 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:29.190270    4441 generic.go:356] "Generic (PLEG): container finished" podID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerID="2747861dfb5d59b5b0a4d4762f4d0b43e3efe8500b2327e4bfed51c1ba01785e" exitCode=0
Oct 30 00:15:29.190439 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:29.190315    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"
Oct 30 00:15:29.190439 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:29.190349    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" event={"ID":"e93d939d-d744-4a1b-a05c-744e65d507cb","Type":"ContainerDied","Data":"2747861dfb5d59b5b0a4d4762f4d0b43e3efe8500b2327e4bfed51c1ba01785e"}
Oct 30 00:15:29.190439 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:29.190399    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4" event={"ID":"e93d939d-d744-4a1b-a05c-744e65d507cb","Type":"ContainerDied","Data":"453607516fceddb4ca189b9a57c19805e117752a5c8c8e13998183ccbfa2f598"}
Oct 30 00:15:29.190439 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:29.190421    4441 scope.go:122] "RemoveContainer" containerID="2747861dfb5d59b5b0a4d4762f4d0b43e3efe8500b2327e4bfed51c1ba01785e"
Oct 30 00:15:29.199636 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:29.199619    4441 scope.go:122] "RemoveContainer" containerID="2da2511da3324aa88b750875c27fb9511edccf40adf38f53a2704cab20aaed39"
Oct 30 00:15:29.208297 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:29.208280    4441 scope.go:122] "RemoveContainer" containerID="2747861dfb5d59b5b0a4d4762f4d0b43e3efe8500b2327e4bfed51c1ba01785e"
Oct 30 00:15:29.208582 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:29.208559    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2747861dfb5d59b5b0a4d4762f4d0b43e3efe8500b2327e4bfed51c1ba01785e\": container with ID starting with 2747861dfb5d59b5b0a4d4762f4d0b43e3efe8500b2327e4bfed51c1ba01785e not found: ID does not exist" containerID="2747861dfb5d59b5b0a4d4762f4d0b43e3efe8500b2327e4bfed51c1ba01785e"
Oct 30 00:15:29.208652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:29.208589    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2747861dfb5d59b5b0a4d4762f4d0b43e3efe8500b2327e4bfed51c1ba01785e"} err="failed to get container status \"2747861dfb5d59b5b0a4d4762f4d0b43e3efe8500b2327e4bfed51c1ba01785e\": rpc error: code = NotFound desc = could not find container \"2747861dfb5d59b5b0a4d4762f4d0b43e3efe8500b2327e4bfed51c1ba01785e\": container with ID starting with 2747861dfb5d59b5b0a4d4762f4d0b43e3efe8500b2327e4bfed51c1ba01785e not found: ID does not exist"
Oct 30 00:15:29.208652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:29.208606    4441 scope.go:122] "RemoveContainer" containerID="2da2511da3324aa88b750875c27fb9511edccf40adf38f53a2704cab20aaed39"
Oct 30 00:15:29.208837 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:29.208822    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da2511da3324aa88b750875c27fb9511edccf40adf38f53a2704cab20aaed39\": container with ID starting with 2da2511da3324aa88b750875c27fb9511edccf40adf38f53a2704cab20aaed39 not found: ID does not exist" containerID="2da2511da3324aa88b750875c27fb9511edccf40adf38f53a2704cab20aaed39"
Oct 30 00:15:29.208871 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:29.208840    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da2511da3324aa88b750875c27fb9511edccf40adf38f53a2704cab20aaed39"} err="failed to get container status \"2da2511da3324aa88b750875c27fb9511edccf40adf38f53a2704cab20aaed39\": rpc error: code = NotFound desc = could not find container \"2da2511da3324aa88b750875c27fb9511edccf40adf38f53a2704cab20aaed39\": container with ID starting with 2da2511da3324aa88b750875c27fb9511edccf40adf38f53a2704cab20aaed39 not found: ID does not exist"
Oct 30 00:15:29.262399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:29.262361    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"]
Oct 30 00:15:29.273898 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:29.273868    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8ffe924e-gw-openshift-default-5644dc6769-2qdg4"]
Oct 30 00:15:30.076623 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.076580    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:30.089309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.089284    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:30.197238 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.197211    4441 generic.go:356] "Generic (PLEG): container finished" podID="92f451d6-9227-4ed4-9301-3a2565e4e479" containerID="291a0ea4530c205ca6ab7e9e668f0e813ae834cb1d93d8378144e858ba8d0dc0" exitCode=60
Oct 30 00:15:30.197698 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.197276    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz" event={"ID":"92f451d6-9227-4ed4-9301-3a2565e4e479","Type":"ContainerDied","Data":"291a0ea4530c205ca6ab7e9e668f0e813ae834cb1d93d8378144e858ba8d0dc0"}
Oct 30 00:15:30.197698 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.197340    4441 scope.go:122] "RemoveContainer" containerID="6743082107dd227cf0e3edaea7b333cc0c888b61e12f3fc270f0789d7e540572"
Oct 30 00:15:30.197820 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.197712    4441 scope.go:122] "RemoveContainer" containerID="291a0ea4530c205ca6ab7e9e668f0e813ae834cb1d93d8378144e858ba8d0dc0"
Oct 30 00:15:30.198010 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:30.197988    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888655-bxdjz_odh-ai-gateway-infra(92f451d6-9227-4ed4-9301-3a2565e4e479)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz" podUID="92f451d6-9227-4ed4-9301-3a2565e4e479"
Oct 30 00:15:30.210193 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.210165    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:15:30.236530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.236496    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"]
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.527797    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"]
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.528748    4441 cpu_manager.go:414] "containerMap: removing container" podUID="de3547ac-b71e-4dd3-9ccc-ece14a199576" containerName="cleanup"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.528769    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3547ac-b71e-4dd3-9ccc-ece14a199576" containerName="cleanup"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.528787    4441 cpu_manager.go:414] "containerMap: removing container" podUID="de3547ac-b71e-4dd3-9ccc-ece14a199576" containerName="cleanup"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.528794    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3547ac-b71e-4dd3-9ccc-ece14a199576" containerName="cleanup"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.528810    4441 cpu_manager.go:414] "containerMap: removing container" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerName="istio-proxy"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.528819    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerName="istio-proxy"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.528863    4441 cpu_manager.go:414] "containerMap: removing container" podUID="de3547ac-b71e-4dd3-9ccc-ece14a199576" containerName="cleanup"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.528870    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3547ac-b71e-4dd3-9ccc-ece14a199576" containerName="cleanup"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.528889    4441 cpu_manager.go:414] "containerMap: removing container" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerName="istio-proxy"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.528896    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerName="istio-proxy"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.529123    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3547ac-b71e-4dd3-9ccc-ece14a199576" containerName="cleanup"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.529135    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3547ac-b71e-4dd3-9ccc-ece14a199576" containerName="cleanup"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.529143    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerName="istio-proxy"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.529161    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3547ac-b71e-4dd3-9ccc-ece14a199576" containerName="cleanup"
Oct 30 00:15:30.529534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.529177    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" containerName="istio-proxy"
Oct 30 00:15:30.533698 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.533674    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"
Oct 30 00:15:30.547079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.547047    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"]
Oct 30 00:15:30.549143 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.549119    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-564c8f\""
Oct 30 00:15:30.604424 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.604387    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a47c65-8652-4da3-a02d-961c57ff4d34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n\" (UID: \"69a47c65-8652-4da3-a02d-961c57ff4d34\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"
Oct 30 00:15:30.604424 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.604433    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6jv\" (UniqueName: \"kubernetes.io/projected/69a47c65-8652-4da3-a02d-961c57ff4d34-kube-api-access-fq6jv\") pod \"payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n\" (UID: \"69a47c65-8652-4da3-a02d-961c57ff4d34\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"
Oct 30 00:15:30.608457 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.608431    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93d939d-d744-4a1b-a05c-744e65d507cb" path="/var/lib/kubelet/pods/e93d939d-d744-4a1b-a05c-744e65d507cb/volumes"
Oct 30 00:15:30.706046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.705590    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a47c65-8652-4da3-a02d-961c57ff4d34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n\" (UID: \"69a47c65-8652-4da3-a02d-961c57ff4d34\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"
Oct 30 00:15:30.706046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.705654    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6jv\" (UniqueName: \"kubernetes.io/projected/69a47c65-8652-4da3-a02d-961c57ff4d34-kube-api-access-fq6jv\") pod \"payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n\" (UID: \"69a47c65-8652-4da3-a02d-961c57ff4d34\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"
Oct 30 00:15:30.706875 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.706764    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a47c65-8652-4da3-a02d-961c57ff4d34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n\" (UID: \"69a47c65-8652-4da3-a02d-961c57ff4d34\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"
Oct 30 00:15:30.728161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.728127    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6jv\" (UniqueName: \"kubernetes.io/projected/69a47c65-8652-4da3-a02d-961c57ff4d34-kube-api-access-fq6jv\") pod \"payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n\" (UID: \"69a47c65-8652-4da3-a02d-961c57ff4d34\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"
Oct 30 00:15:30.847083 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:30.846987    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"
Oct 30 00:15:31.144869 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:31.144845    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"]
Oct 30 00:15:31.147157 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:15:31.147125    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a47c65_8652_4da3_a02d_961c57ff4d34.slice/crio-9dce98c63724e4a0a0556f72686676b60de4128d3120609e6cdbce30dd0d5a47 WatchSource:0}: Error finding container 9dce98c63724e4a0a0556f72686676b60de4128d3120609e6cdbce30dd0d5a47: Status 404 returned error can't find the container with id 9dce98c63724e4a0a0556f72686676b60de4128d3120609e6cdbce30dd0d5a47
Oct 30 00:15:31.206617 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:31.206587    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n" event={"ID":"69a47c65-8652-4da3-a02d-961c57ff4d34","Type":"ContainerStarted","Data":"9dce98c63724e4a0a0556f72686676b60de4128d3120609e6cdbce30dd0d5a47"}
Oct 30 00:15:31.338908 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:31.338881    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"
Oct 30 00:15:31.415440 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:31.415352    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f451d6-9227-4ed4-9301-3a2565e4e479-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f451d6-9227-4ed4-9301-3a2565e4e479-maas-api-service-ca\") pod \"92f451d6-9227-4ed4-9301-3a2565e4e479\" (UID: \"92f451d6-9227-4ed4-9301-3a2565e4e479\") "
Oct 30 00:15:31.415440 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:31.415421    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f451d6-9227-4ed4-9301-3a2565e4e479-kube-api-access-8sj27\" (UniqueName: \"kubernetes.io/projected/92f451d6-9227-4ed4-9301-3a2565e4e479-kube-api-access-8sj27\") pod \"92f451d6-9227-4ed4-9301-3a2565e4e479\" (UID: \"92f451d6-9227-4ed4-9301-3a2565e4e479\") "
Oct 30 00:15:31.415708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:31.415683    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f451d6-9227-4ed4-9301-3a2565e4e479-maas-api-service-ca" pod "92f451d6-9227-4ed4-9301-3a2565e4e479" (UID: "92f451d6-9227-4ed4-9301-3a2565e4e479"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:15:31.415827 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:31.415800    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f451d6-9227-4ed4-9301-3a2565e4e479-maas-api-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:31.417501 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:31.417470    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f451d6-9227-4ed4-9301-3a2565e4e479-kube-api-access-8sj27" pod "92f451d6-9227-4ed4-9301-3a2565e4e479" (UID: "92f451d6-9227-4ed4-9301-3a2565e4e479"). InnerVolumeSpecName "kube-api-access-8sj27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:15:31.516925 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:31.516888    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sj27\" (UniqueName: \"kubernetes.io/projected/92f451d6-9227-4ed4-9301-3a2565e4e479-kube-api-access-8sj27\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:15:32.217813 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.217728    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n" event={"ID":"69a47c65-8652-4da3-a02d-961c57ff4d34","Type":"ContainerStarted","Data":"15c3549ae465f2f755e1a21c1bfa1fad653f8334e4228b9ead9194659031c73d"}
Oct 30 00:15:32.217813 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.217798    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"
Oct 30 00:15:32.219132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.219108    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"
Oct 30 00:15:32.219275 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.219140    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz" event={"ID":"92f451d6-9227-4ed4-9301-3a2565e4e479","Type":"ContainerDied","Data":"399090400d1cf689bb643956cab7f3969f7d10e7c254e5345047d02a72e2bf5f"}
Oct 30 00:15:32.219275 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.219195    4441 scope.go:122] "RemoveContainer" containerID="291a0ea4530c205ca6ab7e9e668f0e813ae834cb1d93d8378144e858ba8d0dc0"
Oct 30 00:15:32.258533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.258478    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n" podStartSLOduration=2.258463937 podStartE2EDuration="2.258463937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:32.256555596 +0000 UTC m=+641.781849486" watchObservedRunningTime="2026-10-30 00:15:32.258463937 +0000 UTC m=+641.783757811"
Oct 30 00:15:32.292883 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.292852    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"]
Oct 30 00:15:32.296846 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.296820    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888655-bxdjz"]
Oct 30 00:15:32.394893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.394864    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"]
Oct 30 00:15:32.395337 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.395324    4441 cpu_manager.go:414] "containerMap: removing container" podUID="92f451d6-9227-4ed4-9301-3a2565e4e479" containerName="cleanup"
Oct 30 00:15:32.395381 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.395338    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f451d6-9227-4ed4-9301-3a2565e4e479" containerName="cleanup"
Oct 30 00:15:32.395381 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.395359    4441 cpu_manager.go:414] "containerMap: removing container" podUID="92f451d6-9227-4ed4-9301-3a2565e4e479" containerName="cleanup"
Oct 30 00:15:32.395381 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.395364    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f451d6-9227-4ed4-9301-3a2565e4e479" containerName="cleanup"
Oct 30 00:15:32.395477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.395446    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f451d6-9227-4ed4-9301-3a2565e4e479" containerName="cleanup"
Oct 30 00:15:32.395477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.395455    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f451d6-9227-4ed4-9301-3a2565e4e479" containerName="cleanup"
Oct 30 00:15:32.398810 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.398788    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:32.414836 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.414809    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-02665b-serving-cert\""
Oct 30 00:15:32.417638 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.417615    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"]
Oct 30 00:15:32.527069 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.527007    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzqxm\" (UniqueName: \"kubernetes.io/projected/abdcfa70-8788-4f09-b420-7a1aa2e03314-kube-api-access-gzqxm\") pod \"maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:32.527249 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.527080    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdcfa70-8788-4f09-b420-7a1aa2e03314-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:32.527299 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.527270    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-metrics-certs\") pod \"maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:32.527337 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.527311    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-maas-api-tls\") pod \"maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:32.608702 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.608672    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f451d6-9227-4ed4-9301-3a2565e4e479" path="/var/lib/kubelet/pods/92f451d6-9227-4ed4-9301-3a2565e4e479/volumes"
Oct 30 00:15:32.628272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.628246    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzqxm\" (UniqueName: \"kubernetes.io/projected/abdcfa70-8788-4f09-b420-7a1aa2e03314-kube-api-access-gzqxm\") pod \"maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:32.628444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.628284    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdcfa70-8788-4f09-b420-7a1aa2e03314-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:32.628444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.628391    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-metrics-certs\") pod \"maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:32.628444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.628417    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-maas-api-tls\") pod \"maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:32.628960 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.628934    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdcfa70-8788-4f09-b420-7a1aa2e03314-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:32.630864 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.630845    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-metrics-certs\") pod \"maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:32.631062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.631008    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-maas-api-tls\") pod \"maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:32.655495 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.655464    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzqxm\" (UniqueName: \"kubernetes.io/projected/abdcfa70-8788-4f09-b420-7a1aa2e03314-kube-api-access-gzqxm\") pod \"maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:32.710747 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:32.710713    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:33.017802 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:33.017770    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"]
Oct 30 00:15:33.019960 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:15:33.019931    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabdcfa70_8788_4f09_b420_7a1aa2e03314.slice/crio-e694dba02bb6e92a3512ce821b57547525a4f2f0be63c1dabd65a2397c4245cb WatchSource:0}: Error finding container e694dba02bb6e92a3512ce821b57547525a4f2f0be63c1dabd65a2397c4245cb: Status 404 returned error can't find the container with id e694dba02bb6e92a3512ce821b57547525a4f2f0be63c1dabd65a2397c4245cb
Oct 30 00:15:33.233897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:33.233861    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t" event={"ID":"abdcfa70-8788-4f09-b420-7a1aa2e03314","Type":"ContainerStarted","Data":"e694dba02bb6e92a3512ce821b57547525a4f2f0be63c1dabd65a2397c4245cb"}
Oct 30 00:15:33.898564 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:15:33.898522    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:15:34.241967 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:34.241928    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t" event={"ID":"abdcfa70-8788-4f09-b420-7a1aa2e03314","Type":"ContainerStarted","Data":"a4b86677ef40e887131d7e29d98c7f1c24df71ba53218af4a5b67ae2f12693b5"}
Oct 30 00:15:34.242510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:34.242129    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:34.282218 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:34.282171    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t" podStartSLOduration=1.990103994 podStartE2EDuration="2.282155271s" totalImagesPullingTime="292.051277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:34.282039823 +0000 UTC m=+643.807333695" watchObservedRunningTime="2026-10-30 00:15:34.282155271 +0000 UTC m=+643.807449143"
Oct 30 00:15:36.160864 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:36.160835    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"
Oct 30 00:15:38.179748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:38.179718    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"
Oct 30 00:15:40.267933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:40.267901    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:15:42.510491 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:42.510452    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:15:43.252488 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:43.251657    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"
Oct 30 00:15:52.157240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:52.157192    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:15:52.157711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:52.157276    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:15:53.397397 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:53.397363    4441 generic.go:356] "Generic (PLEG): container finished" podID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerID="a25a1c66f47386d6aa90a9c7ba8dd7a6bbcb697db8ec209a467a7cc59eea78f0" exitCode=0
Oct 30 00:15:53.397861 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:53.397435    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" event={"ID":"377ffe8e-cc09-4c8b-a25d-c2e486236de5","Type":"ContainerDied","Data":"a25a1c66f47386d6aa90a9c7ba8dd7a6bbcb697db8ec209a467a7cc59eea78f0"}
Oct 30 00:15:53.397913 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:53.397864    4441 scope.go:122] "RemoveContainer" containerID="a25a1c66f47386d6aa90a9c7ba8dd7a6bbcb697db8ec209a467a7cc59eea78f0"
Oct 30 00:15:53.399239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:53.399214    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:53.399293 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:53.399284    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:53.399327 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:53.399318    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:15:54.405700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:54.405665    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" event={"ID":"377ffe8e-cc09-4c8b-a25d-c2e486236de5","Type":"ContainerStarted","Data":"425187aad361de855323a9f3b522ef3d57b0e1233ac6c4b1653b706fe6ae8a9c"}
Oct 30 00:15:54.703109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:54.703019    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:54.703262 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:54.703193    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:15:54.703262 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:54.703242    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:15:55.703716 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:55.703685    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:15:55.704225 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:55.704200    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:15:56.703420 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:56.703331    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:15:56.703420 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:56.703383    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:15:57.646982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:57.646945    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:15:57.707074 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:57.703569    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:15:57.707074 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:57.703635    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:15:58.211232 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:58.211197    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:15:58.211688 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:58.211469    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:15:58.704878 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:58.704837    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:15:58.705499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:58.704913    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:15:59.793569 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:59.793534    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:59.794271 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:59.794238    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:15:59.795934 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:15:59.795909    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:16:01.206320 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.206280    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"]
Oct 30 00:16:01.206989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.206972    4441 cpu_manager.go:414] "containerMap: removing container" podUID="92f451d6-9227-4ed4-9301-3a2565e4e479" containerName="cleanup"
Oct 30 00:16:01.207102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.206992    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f451d6-9227-4ed4-9301-3a2565e4e479" containerName="cleanup"
Oct 30 00:16:01.207183 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.207171    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f451d6-9227-4ed4-9301-3a2565e4e479" containerName="cleanup"
Oct 30 00:16:01.209943 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.209922    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.229996 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.229959    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"]
Oct 30 00:16:01.242067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.235959    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-02665b\"/\"kube-root-ca.crt\""
Oct 30 00:16:01.242067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.236198    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-02665b\"/\"simulator-w4-02665b-kserve-self-signed-certs\""
Oct 30 00:16:01.242067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.236276    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-02665b\"/\"openshift-service-ca.crt\""
Oct 30 00:16:01.242067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.236362    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-02665b\"/\"default-dockercfg-tksv4\""
Oct 30 00:16:01.342915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.342836    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-model-cache\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.342915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.342883    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8txzf\" (UniqueName: \"kubernetes.io/projected/92906908-16f4-45e4-8177-1b53992469a5-kube-api-access-8txzf\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.343130 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.343096    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-dshm\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.343172 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.343153    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-home\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.343263 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.343233    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92906908-16f4-45e4-8177-1b53992469a5-tls-certs\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.343298 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.343287    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-tmp-dir\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.444800 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.444709    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-dshm\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.444800 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.444758    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-home\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.444800 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.444774    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92906908-16f4-45e4-8177-1b53992469a5-tls-certs\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.444800 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.444791    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-tmp-dir\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.445168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.444884    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-model-cache\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.445168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.444934    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8txzf\" (UniqueName: \"kubernetes.io/projected/92906908-16f4-45e4-8177-1b53992469a5-kube-api-access-8txzf\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.445311 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.445286    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-home\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.445396 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.445316    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-tmp-dir\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.445555 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.445532    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-model-cache\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.447442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.447418    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-dshm\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.447728 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.447706    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92906908-16f4-45e4-8177-1b53992469a5-tls-certs\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.468778 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.468750    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8txzf\" (UniqueName: \"kubernetes.io/projected/92906908-16f4-45e4-8177-1b53992469a5-kube-api-access-8txzf\") pod \"simulator-w4-02665b-kserve-d4fcc57cb-kn8d9\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") " pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.488903 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.488865    4441 generic.go:356] "Generic (PLEG): container finished" podID="daae17d4-1319-4ef7-adb7-a013358915db" containerID="9b02c56d3aad4dd1c1ec9d2b66eed5a6acf66073ad5f732b3a4452bde113c547" exitCode=0
Oct 30 00:16:01.489094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.488946    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" event={"ID":"daae17d4-1319-4ef7-adb7-a013358915db","Type":"ContainerDied","Data":"9b02c56d3aad4dd1c1ec9d2b66eed5a6acf66073ad5f732b3a4452bde113c547"}
Oct 30 00:16:01.489662 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.489638    4441 scope.go:122] "RemoveContainer" containerID="9b02c56d3aad4dd1c1ec9d2b66eed5a6acf66073ad5f732b3a4452bde113c547"
Oct 30 00:16:01.492609 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.491571    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:16:01.492609 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.491667    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:16:01.492609 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.491732    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:16:01.552321 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.552004    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:01.889576 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:01.889336    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"]
Oct 30 00:16:01.895243 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:16:01.895199    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92906908_16f4_45e4_8177_1b53992469a5.slice/crio-c0869c73466b5c7285ca2af693d7b1c5073f0df0ec56ccffea9aa906120cfea0 WatchSource:0}: Error finding container c0869c73466b5c7285ca2af693d7b1c5073f0df0ec56ccffea9aa906120cfea0: Status 404 returned error can't find the container with id c0869c73466b5c7285ca2af693d7b1c5073f0df0ec56ccffea9aa906120cfea0
Oct 30 00:16:02.495660 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:02.495616    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" event={"ID":"92906908-16f4-45e4-8177-1b53992469a5","Type":"ContainerStarted","Data":"bd04ecaccc0e151cd7d0602eb86388fa082c9ee85a108c2ef6ac99d4b6a9cdda"}
Oct 30 00:16:02.496189 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:02.495669    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" event={"ID":"92906908-16f4-45e4-8177-1b53992469a5","Type":"ContainerStarted","Data":"c0869c73466b5c7285ca2af693d7b1c5073f0df0ec56ccffea9aa906120cfea0"}
Oct 30 00:16:02.497783 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:02.497751    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" event={"ID":"daae17d4-1319-4ef7-adb7-a013358915db","Type":"ContainerStarted","Data":"bd46f99662b1cd397cce54760b890fbab9b0fedb7fe86f1a7264e89df240f774"}
Oct 30 00:16:02.547119 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:02.536474    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podStartSLOduration=1.536452999 podStartE2EDuration="1.536452999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:02.534516942 +0000 UTC m=+672.059810819" watchObservedRunningTime="2026-10-30 00:16:02.536452999 +0000 UTC m=+672.061746877"
Oct 30 00:16:02.552773 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:02.552745    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:02.959510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:02.959423    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:16:02.959668 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:02.959644    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:16:02.959731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:02.959704    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:16:03.936114 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:16:03.926257    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:16:03.960106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:03.959319    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:16:03.960106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:03.959379    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:16:04.959720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:04.959682    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:16:04.960115 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:04.959741    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:16:05.959762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:05.959726    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:16:05.960286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:05.959793    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:16:06.959310 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:06.959274    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:16:06.959531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:06.959354    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:16:07.963285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:07.963253    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:16:07.963776 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:07.963626    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:16:07.964560 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:07.964534    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:16:11.553248 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:11.553213    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:11.566782 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:11.566757    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:12.565059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:12.565016    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:16:15.929656 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:15.929616    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:16:20.952915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:20.952875    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"]
Oct 30 00:16:20.956002 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:20.955978    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:20.984840 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:20.984817    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b1c718\"/\"openshift-service-ca.crt\""
Oct 30 00:16:20.985007 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:20.984862    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b1c718\"/\"default-dockercfg-q56cf\""
Oct 30 00:16:20.985007 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:20.984862    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b1c718\"/\"kube-root-ca.crt\""
Oct 30 00:16:20.985007 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:20.984884    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b1c718\"/\"premium-w5-b1c718-kserve-self-signed-certs\""
Oct 30 00:16:20.987426 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:20.987403    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"]
Oct 30 00:16:21.011720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.011688    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-tmp-dir\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.011837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.011737    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-home\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.011942 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.011917    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-model-cache\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.012009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.011951    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736f577a-2a21-4448-8c1e-bb71782053ec-tls-certs\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.012114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.012043    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-dshm\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.012114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.012078    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr9db\" (UniqueName: \"kubernetes.io/projected/736f577a-2a21-4448-8c1e-bb71782053ec-kube-api-access-rr9db\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.113399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.113356    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-tmp-dir\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.113399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.113403    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-home\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.113624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.113470    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-model-cache\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.113624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.113492    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736f577a-2a21-4448-8c1e-bb71782053ec-tls-certs\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.113624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.113528    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-dshm\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.113624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.113567    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr9db\" (UniqueName: \"kubernetes.io/projected/736f577a-2a21-4448-8c1e-bb71782053ec-kube-api-access-rr9db\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.113962 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.113940    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-tmp-dir\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.114058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.114039    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-home\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.114247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.114225    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-model-cache\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.115910 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.115887    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-dshm\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.116081 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.116063    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736f577a-2a21-4448-8c1e-bb71782053ec-tls-certs\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.135612 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.135584    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr9db\" (UniqueName: \"kubernetes.io/projected/736f577a-2a21-4448-8c1e-bb71782053ec-kube-api-access-rr9db\") pod \"premium-w5-b1c718-kserve-846d8676d7-9z4bw\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") " pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.271625 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.271592    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:21.570598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.570569    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"]
Oct 30 00:16:21.571281 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:16:21.571251    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod736f577a_2a21_4448_8c1e_bb71782053ec.slice/crio-a6614b32bec3823f454b3ad10b3ea14929297b157096b7d8f0ff17b200024d05 WatchSource:0}: Error finding container a6614b32bec3823f454b3ad10b3ea14929297b157096b7d8f0ff17b200024d05: Status 404 returned error can't find the container with id a6614b32bec3823f454b3ad10b3ea14929297b157096b7d8f0ff17b200024d05
Oct 30 00:16:21.607891 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:21.607867    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" event={"ID":"736f577a-2a21-4448-8c1e-bb71782053ec","Type":"ContainerStarted","Data":"a6614b32bec3823f454b3ad10b3ea14929297b157096b7d8f0ff17b200024d05"}
Oct 30 00:16:22.613853 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:22.613820    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" event={"ID":"736f577a-2a21-4448-8c1e-bb71782053ec","Type":"ContainerStarted","Data":"ae0c003b1e09213693848ae6216c287f8e425fd1d61c3a0c70e61aeecfe50d0a"}
Oct 30 00:16:22.656046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:22.655989    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podStartSLOduration=2.655975887 podStartE2EDuration="2.655975887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:22.653708105 +0000 UTC m=+692.179001983" watchObservedRunningTime="2026-10-30 00:16:22.655975887 +0000 UTC m=+692.181269764"
Oct 30 00:16:23.272105 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:23.272073    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:31.272556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:31.272524    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:31.285064 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:31.285036    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:31.685304 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:31.685225    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:16:33.898770 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:16:33.898730    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:16:39.204474 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:39.204436    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:16:42.775574 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:42.775538    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"]
Oct 30 00:16:42.778467 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:42.778445    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:42.804211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:42.804187    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-322d20\"/\"openshift-service-ca.crt\""
Oct 30 00:16:42.804332 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:42.804270    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-322d20\"/\"kube-root-ca.crt\""
Oct 30 00:16:42.804332 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:42.804282    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-322d20\"/\"default-dockercfg-kk5f8\""
Oct 30 00:16:42.804620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:42.804600    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-322d20\"/\"premium-w2-322d20-kserve-self-signed-certs\""
Oct 30 00:16:42.806866 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:42.806844    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"]
Oct 30 00:16:42.972126 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:42.972088    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-home\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:42.972126 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:42.972130    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-model-cache\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:42.972380 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:42.972271    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj6j7\" (UniqueName: \"kubernetes.io/projected/929cb9ab-721f-4d60-b304-50aa1bab3573-kube-api-access-nj6j7\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:42.972440 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:42.972422    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-tmp-dir\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:42.972497 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:42.972445    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929cb9ab-721f-4d60-b304-50aa1bab3573-tls-certs\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:42.972497 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:42.972481    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-dshm\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.073427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.073352    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-dshm\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.073427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.073394    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-home\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.073427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.073413    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-model-cache\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.073667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.073470    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj6j7\" (UniqueName: \"kubernetes.io/projected/929cb9ab-721f-4d60-b304-50aa1bab3573-kube-api-access-nj6j7\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.073667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.073546    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-tmp-dir\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.073667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.073559    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929cb9ab-721f-4d60-b304-50aa1bab3573-tls-certs\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.073895 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.073871    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-model-cache\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.073957 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.073904    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-home\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.073957 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.073939    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-tmp-dir\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.075533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.075510    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-dshm\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.075821 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.075804    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929cb9ab-721f-4d60-b304-50aa1bab3573-tls-certs\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.096441 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.096415    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj6j7\" (UniqueName: \"kubernetes.io/projected/929cb9ab-721f-4d60-b304-50aa1bab3573-kube-api-access-nj6j7\") pod \"premium-w2-322d20-kserve-86dfb66bc9-5qrlw\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") " pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.390351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.390273    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:43.689067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.689037    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"]
Oct 30 00:16:43.690721 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:16:43.690694    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod929cb9ab_721f_4d60_b304_50aa1bab3573.slice/crio-29a9c34a01f952945d22556cbb1d9ad0638f246daf42c3c8c9d354a19060a624 WatchSource:0}: Error finding container 29a9c34a01f952945d22556cbb1d9ad0638f246daf42c3c8c9d354a19060a624: Status 404 returned error can't find the container with id 29a9c34a01f952945d22556cbb1d9ad0638f246daf42c3c8c9d354a19060a624
Oct 30 00:16:43.734627 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:43.734600    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" event={"ID":"929cb9ab-721f-4d60-b304-50aa1bab3573","Type":"ContainerStarted","Data":"29a9c34a01f952945d22556cbb1d9ad0638f246daf42c3c8c9d354a19060a624"}
Oct 30 00:16:44.742394 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:44.742363    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" event={"ID":"929cb9ab-721f-4d60-b304-50aa1bab3573","Type":"ContainerStarted","Data":"3611ca84f1d8ba700823592149bfb4134e4450f8baa8973da4450306c06b7398"}
Oct 30 00:16:44.784640 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:44.784580    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podStartSLOduration=2.784560054 podStartE2EDuration="2.784560054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:44.781960743 +0000 UTC m=+714.307254616" watchObservedRunningTime="2026-10-30 00:16:44.784560054 +0000 UTC m=+714.309853932"
Oct 30 00:16:45.391343 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:45.391312    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:51.860792 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:51.860758    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"]
Oct 30 00:16:51.864528 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:51.864496    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:51.881851 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:51.881823    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"]
Oct 30 00:16:51.889659 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:51.889632    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-564c8f\"/\"openshift-service-ca.crt\""
Oct 30 00:16:51.889858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:51.889677    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-564c8f\"/\"kube-root-ca.crt\""
Oct 30 00:16:51.891137 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:51.891071    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-564c8f\"/\"premium-w3-564c8f-kserve-self-signed-certs\""
Oct 30 00:16:51.891137 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:51.891072    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-564c8f\"/\"default-dockercfg-jqzgw\""
Oct 30 00:16:51.975173 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:51.975136    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-model-cache\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:51.975355 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:51.975290    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwff\" (UniqueName: \"kubernetes.io/projected/39b25719-53ed-4967-8d2a-153ed54b739c-kube-api-access-hzwff\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:51.975405 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:51.975362    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-home\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:51.975464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:51.975438    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b25719-53ed-4967-8d2a-153ed54b739c-tls-certs\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:51.975504 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:51.975488    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-tmp-dir\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:51.975598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:51.975582    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-dshm\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.076732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.076694    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-dshm\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.076913 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.076783    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-model-cache\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.076913 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.076844    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwff\" (UniqueName: \"kubernetes.io/projected/39b25719-53ed-4967-8d2a-153ed54b739c-kube-api-access-hzwff\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.076913 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.076880    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-home\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.076913 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.076896    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b25719-53ed-4967-8d2a-153ed54b739c-tls-certs\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.077167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.076916    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-tmp-dir\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.077317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.077281    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-model-cache\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.077513 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.077488    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-home\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.077591 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.077515    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-tmp-dir\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.079453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.079427    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-dshm\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.079572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.079540    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b25719-53ed-4967-8d2a-153ed54b739c-tls-certs\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.099674 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.099641    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwff\" (UniqueName: \"kubernetes.io/projected/39b25719-53ed-4967-8d2a-153ed54b739c-kube-api-access-hzwff\") pod \"premium-w3-564c8f-kserve-57b794fcb7-j2hwn\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") " pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.184351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.184274    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:52.487722 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.487689    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"]
Oct 30 00:16:52.488054 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:16:52.488016    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39b25719_53ed_4967_8d2a_153ed54b739c.slice/crio-c1547a61a125aab2b005ca55dc31db4c38000872f6dc4d5ff22911ea56bf1d53 WatchSource:0}: Error finding container c1547a61a125aab2b005ca55dc31db4c38000872f6dc4d5ff22911ea56bf1d53: Status 404 returned error can't find the container with id c1547a61a125aab2b005ca55dc31db4c38000872f6dc4d5ff22911ea56bf1d53
Oct 30 00:16:52.791476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.791389    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" event={"ID":"39b25719-53ed-4967-8d2a-153ed54b739c","Type":"ContainerStarted","Data":"2790d95ed7afed6417752a563fd974da2076592674ac45dd4f8b0809413209ac"}
Oct 30 00:16:52.791476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.791436    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" event={"ID":"39b25719-53ed-4967-8d2a-153ed54b739c","Type":"ContainerStarted","Data":"c1547a61a125aab2b005ca55dc31db4c38000872f6dc4d5ff22911ea56bf1d53"}
Oct 30 00:16:52.833418 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:52.833355    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podStartSLOduration=1.8333371569999999 podStartE2EDuration="1.833337157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:52.83013602 +0000 UTC m=+722.355429897" watchObservedRunningTime="2026-10-30 00:16:52.833337157 +0000 UTC m=+722.358631038"
Oct 30 00:16:53.184819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:53.184736    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:16:53.390644 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:53.390601    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:53.403605 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:53.403576    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:53.808534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:53.808508    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:16:58.787937 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.787903    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"]
Oct 30 00:16:58.792011 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.791991    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.813075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.813044    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"]
Oct 30 00:16:58.848763 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.848736    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.848911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.848777    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.848911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.848800    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.848994 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.848935    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp474\" (UniqueName: \"kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-kube-api-access-bp474\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.849042 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.848995    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.849079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.849053    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.849113 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.849089    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.849167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.849118    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.849286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.849263    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.949976 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.949938    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp474\" (UniqueName: \"kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-kube-api-access-bp474\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.950171 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.949985    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.950171 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.950018    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.950171 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.950077    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.950171 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.950114    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.950389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.950191    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.950389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.950276    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.950389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.950329    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.950389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.950379    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.950613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.950507    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.950613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.950579    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.950754 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.950731    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.950791 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.950763    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.950994 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.950967    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.952564 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.952542    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.953120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.953098    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.973772 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.973744    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:58.973890 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:58.973830    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp474\" (UniqueName: \"kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-kube-api-access-bp474\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:59.109298 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:59.109216    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:16:59.422972 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:59.422945    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"]
Oct 30 00:16:59.424650 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:16:59.424616    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ffa28d_7b18_46c3_99e0_3783316b0dc6.slice/crio-b976a51639f648adce8a27794a704e956fec589460a601dedca1a2f083afb87b WatchSource:0}: Error finding container b976a51639f648adce8a27794a704e956fec589460a601dedca1a2f083afb87b: Status 404 returned error can't find the container with id b976a51639f648adce8a27794a704e956fec589460a601dedca1a2f083afb87b
Oct 30 00:16:59.426976 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:59.426943    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:16:59.427096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:59.427008    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:16:59.427096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:59.427058    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:16:59.831711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:59.831676    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" event={"ID":"16ffa28d-7b18-46c3-99e0-3783316b0dc6","Type":"ContainerStarted","Data":"896c066a342cba7d8ecb39f89394b6a1cb506fba6107a2caaceb2dc30b6ccdef"}
Oct 30 00:16:59.832154 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:59.831732    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" event={"ID":"16ffa28d-7b18-46c3-99e0-3783316b0dc6","Type":"ContainerStarted","Data":"b976a51639f648adce8a27794a704e956fec589460a601dedca1a2f083afb87b"}
Oct 30 00:16:59.905142 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:16:59.905083    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" podStartSLOduration=1.9050671270000001 podStartE2EDuration="1.905067127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:59.885897821 +0000 UTC m=+729.411191715" watchObservedRunningTime="2026-10-30 00:16:59.905067127 +0000 UTC m=+729.430361004"
Oct 30 00:17:00.109875 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:00.109790    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:17:00.111078 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:00.111050    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 00:17:00.111223 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:00.111105    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" podUID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 00:17:01.110111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:01.110074    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 00:17:01.110478 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:01.110133    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" podUID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 00:17:02.110608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:02.110572    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 00:17:02.111020 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:02.110638    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" podUID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 00:17:02.185009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:02.184979    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:17:02.197732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:02.197709    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:17:02.887279 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:02.887251    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:17:03.109996 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:03.109960    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 00:17:03.110207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:03.110079    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" podUID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 00:17:03.900652 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:17:03.900619    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:17:04.109843 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:04.109812    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 00:17:04.110009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:04.109875    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" podUID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 00:17:04.360780 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:04.360738    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:17:05.109735 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:05.109699    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 00:17:05.110170 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:05.109755    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" podUID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 00:17:06.287140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:06.286889    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:17:06.287848 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:06.287826    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:17:06.288823 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:06.288795    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:17:19.769817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:19.769779    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"]
Oct 30 00:17:19.774643 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:19.774619    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:19.791882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:19.791859    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"]
Oct 30 00:17:19.800813 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:19.800790    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b1c718\"/\"distinct-w5-b1c718-kserve-self-signed-certs\""
Oct 30 00:17:19.930689 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:19.930653    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-model-cache\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:19.930689 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:19.930689    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tmp-dir\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:19.930934 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:19.930723    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-dshm\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:19.930934 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:19.930815    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9whp\" (UniqueName: \"kubernetes.io/projected/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-kube-api-access-t9whp\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:19.931084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:19.930963    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-home\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:19.931084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:19.930998    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tls-certs\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.032414 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.032281    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-dshm\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.032414 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.032350    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9whp\" (UniqueName: \"kubernetes.io/projected/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-kube-api-access-t9whp\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.032651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.032529    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-home\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.032651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.032576    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tls-certs\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.032777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.032734    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-model-cache\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.032777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.032762    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tmp-dir\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.032978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.032959    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-home\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.033140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.033121    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-model-cache\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.033236 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.033213    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tmp-dir\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.034599 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.034577    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-dshm\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.035300 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.035280    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tls-certs\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.055037 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.054994    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9whp\" (UniqueName: \"kubernetes.io/projected/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-kube-api-access-t9whp\") pod \"distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.089001 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.088978    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:20.388307 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.388282    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"]
Oct 30 00:17:20.390920 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:17:20.390895    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4829f2a_b4dc_413e_b235_66bc5c0edaaf.slice/crio-18209faf4b08e58d7022db1d1f9acd08d062aa1d98593091d04bc596fe5361fe WatchSource:0}: Error finding container 18209faf4b08e58d7022db1d1f9acd08d062aa1d98593091d04bc596fe5361fe: Status 404 returned error can't find the container with id 18209faf4b08e58d7022db1d1f9acd08d062aa1d98593091d04bc596fe5361fe
Oct 30 00:17:20.995761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.995728    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" event={"ID":"e4829f2a-b4dc-413e-b235-66bc5c0edaaf","Type":"ContainerStarted","Data":"3b5cb819663519d74a834e7f07863712fa4eadfef944ddb760b4a83225a9cb31"}
Oct 30 00:17:20.995761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:20.995768    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" event={"ID":"e4829f2a-b4dc-413e-b235-66bc5c0edaaf","Type":"ContainerStarted","Data":"18209faf4b08e58d7022db1d1f9acd08d062aa1d98593091d04bc596fe5361fe"}
Oct 30 00:17:21.034830 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:21.034550    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podStartSLOduration=2.034535177 podStartE2EDuration="2.034535177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:21.034048841 +0000 UTC m=+750.559342729" watchObservedRunningTime="2026-10-30 00:17:21.034535177 +0000 UTC m=+750.559829057"
Oct 30 00:17:21.089609 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:21.089580    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:30.089594 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:30.089564    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:30.102048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:30.102000    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:30.465423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:30.465341    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:17:31.085316 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:31.085282    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:17:33.904193 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:17:33.904154    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:17:46.855613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:46.855575    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"]
Oct 30 00:17:46.860224 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:46.860203    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:46.886771 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:46.886738    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"e2e-conflict-b-cee02204-kserve-self-signed-certs\""
Oct 30 00:17:46.889829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:46.889805    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"]
Oct 30 00:17:46.972452 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:46.972382    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtxhh\" (UniqueName: \"kubernetes.io/projected/4837c61b-790f-4bc8-ae92-56365684f5d4-kube-api-access-xtxhh\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:46.972618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:46.972574    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-dshm\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:46.972688 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:46.972617    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-tmp-dir\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:46.972767 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:46.972752    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4837c61b-790f-4bc8-ae92-56365684f5d4-tls-certs\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:46.972894 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:46.972875    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-home\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:46.972940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:46.972905    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-model-cache\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.080933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.080896    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-dshm\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.081219 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.081198    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-tmp-dir\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.082039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.081902    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-tmp-dir\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.082237 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.082218    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4837c61b-790f-4bc8-ae92-56365684f5d4-tls-certs\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.082804 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.082780    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-home\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.082919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.082818    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-model-cache\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.082919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.082901    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtxhh\" (UniqueName: \"kubernetes.io/projected/4837c61b-790f-4bc8-ae92-56365684f5d4-kube-api-access-xtxhh\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.083371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.083331    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-model-cache\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.083467 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.083399    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-home\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.086879 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.086855    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4837c61b-790f-4bc8-ae92-56365684f5d4-tls-certs\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.086988 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.086956    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-dshm\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.106279 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.106210    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtxhh\" (UniqueName: \"kubernetes.io/projected/4837c61b-790f-4bc8-ae92-56365684f5d4-kube-api-access-xtxhh\") pod \"e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.176562 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.176524    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:47.483724 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:47.483697    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"]
Oct 30 00:17:47.485322 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:17:47.485297    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4837c61b_790f_4bc8_ae92_56365684f5d4.slice/crio-c01fe16f461c3908efa3c886f4d0c087bfd6e7a5415bef1f9becc0c1a66b4ecd WatchSource:0}: Error finding container c01fe16f461c3908efa3c886f4d0c087bfd6e7a5415bef1f9becc0c1a66b4ecd: Status 404 returned error can't find the container with id c01fe16f461c3908efa3c886f4d0c087bfd6e7a5415bef1f9becc0c1a66b4ecd
Oct 30 00:17:48.175196 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:48.175154    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg" event={"ID":"4837c61b-790f-4bc8-ae92-56365684f5d4","Type":"ContainerStarted","Data":"41b939d0dda494717e33234098d9b46d525dbd09c93ce8ac5404b0378a68bbc3"}
Oct 30 00:17:48.175196 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:48.175200    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg" event={"ID":"4837c61b-790f-4bc8-ae92-56365684f5d4","Type":"ContainerStarted","Data":"c01fe16f461c3908efa3c886f4d0c087bfd6e7a5415bef1f9becc0c1a66b4ecd"}
Oct 30 00:17:48.177443 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:48.177421    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:48.219759 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:48.219698    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg" podStartSLOduration=2.219679491 podStartE2EDuration="2.219679491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:48.217271268 +0000 UTC m=+777.742565160" watchObservedRunningTime="2026-10-30 00:17:48.219679491 +0000 UTC m=+777.744973370"
Oct 30 00:17:49.565788 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.565752    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"]
Oct 30 00:17:49.570246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.570224    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.587119 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.587098    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-322d20-openshift-default-dockercfg-5q64p\""
Oct 30 00:17:49.601482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.601457    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"]
Oct 30 00:17:49.609600 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.609573    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.609761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.609631    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.609761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.609711    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.609881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.609770    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.609933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.609880    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.609933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.609917    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgjtm\" (UniqueName: \"kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-kube-api-access-qgjtm\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.610013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.610001    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30206a75-65df-4bc5-893f-5f4dc29a650c-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.610073 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.610052    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.610129 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.610110    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.711279 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.711244    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.711279 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.711280    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.711499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.711309    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgjtm\" (UniqueName: \"kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-kube-api-access-qgjtm\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.711499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.711383    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30206a75-65df-4bc5-893f-5f4dc29a650c-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.711499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.711432    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.711499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.711461    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.711720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.711598    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.711720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.711639    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.711720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.711667    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.711720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.711673    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.711923 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.711734    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.711923 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.711760    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.712000 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.711971    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.712629 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.712600    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30206a75-65df-4bc5-893f-5f4dc29a650c-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.713862 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.713842    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.714093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.714077    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.733247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.733219    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgjtm\" (UniqueName: \"kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-kube-api-access-qgjtm\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.733357 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.733306    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:49.884960 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:49.884889    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:50.218078 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:50.218049    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"]
Oct 30 00:17:50.219631 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:17:50.219605    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30206a75_65df_4bc5_893f_5f4dc29a650c.slice/crio-62a83d46a4c3481de1e09eb9435a9f41574be998c3850b422aa7190866d6b3a8 WatchSource:0}: Error finding container 62a83d46a4c3481de1e09eb9435a9f41574be998c3850b422aa7190866d6b3a8: Status 404 returned error can't find the container with id 62a83d46a4c3481de1e09eb9435a9f41574be998c3850b422aa7190866d6b3a8
Oct 30 00:17:50.221889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:50.221861    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:17:50.221951 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:50.221924    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:17:50.221987 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:50.221958    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:17:51.195684 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:51.195644    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z" event={"ID":"30206a75-65df-4bc5-893f-5f4dc29a650c","Type":"ContainerStarted","Data":"7f1ef0950c85c2e928dffdea2aaa550deeb13283c7411c2c63e2371f421b21f1"}
Oct 30 00:17:51.196121 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:51.195728    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z" event={"ID":"30206a75-65df-4bc5-893f-5f4dc29a650c","Type":"ContainerStarted","Data":"62a83d46a4c3481de1e09eb9435a9f41574be998c3850b422aa7190866d6b3a8"}
Oct 30 00:17:51.273946 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:51.273880    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z" podStartSLOduration=2.273860775 podStartE2EDuration="2.273860775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:51.260003905 +0000 UTC m=+780.785297806" watchObservedRunningTime="2026-10-30 00:17:51.273860775 +0000 UTC m=+780.799154653"
Oct 30 00:17:51.886066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:51.886012    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:51.887460 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:51.887438    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:17:51.887578 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:51.887494    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z" podUID="30206a75-65df-4bc5-893f-5f4dc29a650c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 00:17:51.959676 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:51.959642    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"]
Oct 30 00:17:51.963907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:51.963887    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:51.973592 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:51.973564    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"]
Oct 30 00:17:51.977749 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:51.977724    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:51.981838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:51.981816    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b1c718-openshift-default-dockercfg-8spcf\""
Oct 30 00:17:51.989734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:51.989705    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"]
Oct 30 00:17:52.002227 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.000476    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"]
Oct 30 00:17:52.058291 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058243    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.058291 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058292    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.058506 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058371    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.058506 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058438    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.058506 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058476    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.058621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058585    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.058662 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058628    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.058704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058661    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.058790 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058764    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.058832 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058814    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmzv5\" (UniqueName: \"kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-kube-api-access-cmzv5\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.058878 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058842    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.058911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058873    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.058911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058905    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0360d680-dc02-4fd2-a4fd-47f584e20527-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.058994 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058923    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnn8n\" (UniqueName: \"kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-kube-api-access-cnn8n\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.058994 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.058945    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.059141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.059122    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.059190 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.059169    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.059248 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.059232    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.160165 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160072    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.160165 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160124    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.160376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160297    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.160438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160400    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.160438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160420    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.160550 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160438    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.160550 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160505    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.160657 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160560    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.160657 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160600    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.160657 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160640    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.160841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160674    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.160841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160732    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.160841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160750    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.160841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160767    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzv5\" (UniqueName: \"kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-kube-api-access-cmzv5\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.160841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160794    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.160841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160804    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.160841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160826    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.161256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160853    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0360d680-dc02-4fd2-a4fd-47f584e20527-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.161256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160877    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnn8n\" (UniqueName: \"kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-kube-api-access-cnn8n\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.161256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.160904    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.161256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.161057    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.161256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.161146    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.161256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.161166    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.161576 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.161380    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.161576 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.161414    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.161576 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.161444    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.161795 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.161777    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.162048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.162009    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0360d680-dc02-4fd2-a4fd-47f584e20527-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.163331 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.163296    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.163459 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.163409    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.163849 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.163825    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.164873 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.164847    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.183993 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.183968    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.184260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.184241    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzv5\" (UniqueName: \"kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-kube-api-access-cmzv5\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.184352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.184336    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.184612 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.184577    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnn8n\" (UniqueName: \"kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-kube-api-access-cnn8n\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.277614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.277580    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:52.292133 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.292111    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:52.666974 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.666939    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"]
Oct 30 00:17:52.670961 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.670930    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:17:52.671080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.671007    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:17:52.671080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.671054    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:17:52.694523 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.694503    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"]
Oct 30 00:17:52.697091 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:17:52.697061    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0360d680_dc02_4fd2_a4fd_47f584e20527.slice/crio-97b6cb72842818af231722599239156c3ad14e816545fa1125ada821ee8c7c61 WatchSource:0}: Error finding container 97b6cb72842818af231722599239156c3ad14e816545fa1125ada821ee8c7c61: Status 404 returned error can't find the container with id 97b6cb72842818af231722599239156c3ad14e816545fa1125ada821ee8c7c61
Oct 30 00:17:52.699616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.699581    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:17:52.699691 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.699663    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:17:52.699734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.699709    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:17:52.886705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.886218    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:17:52.886705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:52.886275    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z" podUID="30206a75-65df-4bc5-893f-5f4dc29a650c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 00:17:53.239050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.219307    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g" event={"ID":"dffac52a-994f-43ce-ad5b-0abf6a73f13a","Type":"ContainerStarted","Data":"36e832d24369d88e6e1dd68d3f0c57ade14751b70188ec1553b045720a9bb97a"}
Oct 30 00:17:53.239050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.219384    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g" event={"ID":"dffac52a-994f-43ce-ad5b-0abf6a73f13a","Type":"ContainerStarted","Data":"b85af8595aa7cf7183202bee6d9495d6acf1d5ba862a460a6e44c306db0d8877"}
Oct 30 00:17:53.239050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.223518    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s" event={"ID":"0360d680-dc02-4fd2-a4fd-47f584e20527","Type":"ContainerStarted","Data":"4ff805a1c6fa32a453fd2739f4498fd8a9251d57189d439530ebfa5fcacc7f12"}
Oct 30 00:17:53.239050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.223565    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s" event={"ID":"0360d680-dc02-4fd2-a4fd-47f584e20527","Type":"ContainerStarted","Data":"97b6cb72842818af231722599239156c3ad14e816545fa1125ada821ee8c7c61"}
Oct 30 00:17:53.267715 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.267636    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g" podStartSLOduration=2.26761815 podStartE2EDuration="2.26761815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:53.265469705 +0000 UTC m=+782.790763595" watchObservedRunningTime="2026-10-30 00:17:53.26761815 +0000 UTC m=+782.792912029"
Oct 30 00:17:53.282050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.278652    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:17:53.282574 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.282537    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:17:53.282640 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.282589    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g" podUID="dffac52a-994f-43ce-ad5b-0abf6a73f13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:17:53.302121 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.301180    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:17:53.302121 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.301316    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 00:17:53.302121 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.301363    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s" podUID="0360d680-dc02-4fd2-a4fd-47f584e20527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 00:17:53.312055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.311356    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s" podStartSLOduration=2.311335728 podStartE2EDuration="2.311335728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:53.307660017 +0000 UTC m=+782.832953897" watchObservedRunningTime="2026-10-30 00:17:53.311335728 +0000 UTC m=+782.836629607"
Oct 30 00:17:53.889049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.886469    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:17:53.889049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:53.886543    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z" podUID="30206a75-65df-4bc5-893f-5f4dc29a650c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 00:17:54.286944 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:54.286459    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:17:54.286944 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:54.286536    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g" podUID="dffac52a-994f-43ce-ad5b-0abf6a73f13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:17:54.298531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:54.298412    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 00:17:54.298531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:54.298473    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s" podUID="0360d680-dc02-4fd2-a4fd-47f584e20527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 00:17:54.902049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:54.886543    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:17:54.902049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:54.886606    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z" podUID="30206a75-65df-4bc5-893f-5f4dc29a650c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 00:17:55.297608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:55.297574    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 00:17:55.298210 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:55.298179    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s" podUID="0360d680-dc02-4fd2-a4fd-47f584e20527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 00:17:55.298961 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:55.298940    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:17:55.299162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:55.299140    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g" podUID="dffac52a-994f-43ce-ad5b-0abf6a73f13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:17:55.886049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:55.885957    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:17:55.886049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:55.886015    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z" podUID="30206a75-65df-4bc5-893f-5f4dc29a650c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 00:17:56.293417 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:56.293389    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:17:56.293645 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:56.293625    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g" podUID="dffac52a-994f-43ce-ad5b-0abf6a73f13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:17:56.294163 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:56.294136    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 00:17:56.294334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:56.294313    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s" podUID="0360d680-dc02-4fd2-a4fd-47f584e20527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 00:17:57.205476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:57.205444    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:57.302540 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:57.302505    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:17:57.302805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:57.302778    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g" podUID="dffac52a-994f-43ce-ad5b-0abf6a73f13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:17:57.303627 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:57.303601    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 00:17:57.305125 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:57.305093    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s" podUID="0360d680-dc02-4fd2-a4fd-47f584e20527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 00:17:57.306475 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:57.306461    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:57.428377 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:57.428335    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:17:57.445621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:57.445584    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:57.447252 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:57.447232    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:57.502052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:57.496384    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:17:58.280049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:58.279404    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:17:58.280049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:58.279468    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g" podUID="dffac52a-994f-43ce-ad5b-0abf6a73f13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:17:58.293055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:58.293006    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 00:17:58.293212 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:58.293071    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s" podUID="0360d680-dc02-4fd2-a4fd-47f584e20527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 00:17:59.278801 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:59.278767    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:17:59.278979 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:59.278818    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g" podUID="dffac52a-994f-43ce-ad5b-0abf6a73f13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:17:59.292705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:59.292677    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 00:17:59.293046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:17:59.292716    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s" podUID="0360d680-dc02-4fd2-a4fd-47f584e20527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 00:18:00.283067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:00.283011    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:18:00.283344 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:00.283324    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:18:00.283852 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:00.283838    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:18:00.296841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:00.296817    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:18:00.297178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:00.297059    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:18:00.297707 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:00.297689    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:18:01.447383 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:01.447348    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:18:02.986698 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:02.986662    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"]
Oct 30 00:18:02.991888 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:02.991865    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.020327 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.020290    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6c45bb\"/\"kube-root-ca.crt\""
Oct 30 00:18:03.020327 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.020316    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6c45bb\"/\"distinct-2-w6-6c45bb-kserve-self-signed-certs\""
Oct 30 00:18:03.020536 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.020414    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6c45bb\"/\"default-dockercfg-z7289\""
Oct 30 00:18:03.020536 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.020456    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6c45bb\"/\"openshift-service-ca.crt\""
Oct 30 00:18:03.024250 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.024225    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"]
Oct 30 00:18:03.071153 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.071121    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-dshm\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.071321 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.071226    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gckx\" (UniqueName: \"kubernetes.io/projected/462c5bbb-db90-4513-a81e-f7514d428449-kube-api-access-9gckx\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.071321 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.071257    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462c5bbb-db90-4513-a81e-f7514d428449-tls-certs\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.071442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.071378    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-tmp-dir\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.071442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.071428    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-model-cache\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.071576 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.071552    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-home\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.175676 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.173457    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-model-cache\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.175676 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.173539    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-home\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.175676 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.173609    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-dshm\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.175676 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.173685    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gckx\" (UniqueName: \"kubernetes.io/projected/462c5bbb-db90-4513-a81e-f7514d428449-kube-api-access-9gckx\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.175676 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.173716    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462c5bbb-db90-4513-a81e-f7514d428449-tls-certs\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.175676 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.173817    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-tmp-dir\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.175676 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.174252    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-tmp-dir\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.175676 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.174502    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-model-cache\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.175676 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.174733    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-home\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.178664 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.177520    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-dshm\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.179174 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.179130    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462c5bbb-db90-4513-a81e-f7514d428449-tls-certs\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.210801 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.210717    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gckx\" (UniqueName: \"kubernetes.io/projected/462c5bbb-db90-4513-a81e-f7514d428449-kube-api-access-9gckx\") pod \"distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.304903 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.304800    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:03.608484 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:03.608454    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"]
Oct 30 00:18:03.610160 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:18:03.610130    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod462c5bbb_db90_4513_a81e_f7514d428449.slice/crio-7e5a6d9093e2e1b11b41ecb6c36b3614cb96ea5552c9880cf5116d7d091a10cd WatchSource:0}: Error finding container 7e5a6d9093e2e1b11b41ecb6c36b3614cb96ea5552c9880cf5116d7d091a10cd: Status 404 returned error can't find the container with id 7e5a6d9093e2e1b11b41ecb6c36b3614cb96ea5552c9880cf5116d7d091a10cd
Oct 30 00:18:03.899405 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:18:03.899324    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:18:04.369497 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:04.369464    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" event={"ID":"462c5bbb-db90-4513-a81e-f7514d428449","Type":"ContainerStarted","Data":"7aef5b118d70a80aff45a1283d2e6d2a7e5ad9b35df02112d4ec080a251d8247"}
Oct 30 00:18:04.370006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:04.369509    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" event={"ID":"462c5bbb-db90-4513-a81e-f7514d428449","Type":"ContainerStarted","Data":"7e5a6d9093e2e1b11b41ecb6c36b3614cb96ea5552c9880cf5116d7d091a10cd"}
Oct 30 00:18:04.409379 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:04.409336    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podStartSLOduration=2.409323691 podStartE2EDuration="2.409323691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:04.408134838 +0000 UTC m=+793.933428709" watchObservedRunningTime="2026-10-30 00:18:04.409323691 +0000 UTC m=+793.934617567"
Oct 30 00:18:05.305323 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:05.305293    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:13.305132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:13.305098    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:13.318529 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:13.318504    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:13.433147 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:13.433109    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:18:26.477542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:26.477507    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:18:31.327848 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.327814    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"]
Oct 30 00:18:31.333622 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.333599    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.343454 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.343428    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"]
Oct 30 00:18:31.348613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.348590    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.353290 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.353247    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7505bd-openshift-default-dockercfg-98hdd\""
Oct 30 00:18:31.354661 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.354634    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"]
Oct 30 00:18:31.372062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.369783    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"]
Oct 30 00:18:31.481246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481210    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.481450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481253    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.481450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481319    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.481450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481333    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.481450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481358    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs7lf\" (UniqueName: \"kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-kube-api-access-fs7lf\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.481645 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481469    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.481645 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481502    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htwls\" (UniqueName: \"kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-kube-api-access-htwls\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.481645 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481537    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.481645 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481631    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.481771 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481708    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.481771 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481732    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.481876 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481779    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.481876 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481841    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.481942 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481883    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d7f0fa-9352-4408-b1a1-cc80af286710-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.481942 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481924    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.482009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.481953    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.482094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.482064    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.482131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.482112    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.583528 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.583440    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.583528 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.583473    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.583528 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.583520    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.583815 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.583549    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.583815 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.583574    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.583815 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.583710    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.583815 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.583767    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.583815 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.583794    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.584106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.583835    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs7lf\" (UniqueName: \"kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-kube-api-access-fs7lf\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.584106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.583898    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.584106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.583924    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htwls\" (UniqueName: \"kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-kube-api-access-htwls\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.584106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.583979    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.584106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584002    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.584106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584013    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.584106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584044    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.584106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584080    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.584518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584115    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.584518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584182    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.584518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584211    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.584518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584211    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.584518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584267    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d7f0fa-9352-4408-b1a1-cc80af286710-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.584518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584313    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.584518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584079    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.584892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584578    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.584892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584754    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.584892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584789    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.584892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584844    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.585148 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.584972    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d7f0fa-9352-4408-b1a1-cc80af286710-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.586467 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.586434    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.586649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.586627    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.587047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.587002    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.587047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.587012    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.606528 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.606499    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.606666 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.606558    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.606731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.606679    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htwls\" (UniqueName: \"kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-kube-api-access-htwls\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.608119 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.608099    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs7lf\" (UniqueName: \"kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-kube-api-access-fs7lf\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.650733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.650698    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:31.675157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.675124    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:31.973329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.973297    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"]
Oct 30 00:18:31.974574 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:18:31.974548    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe79b3ad_1c24_4dde_bc30_23a4264e35f9.slice/crio-c60e696764b8428326b679d4297dd4826d3cb41f248ad3032cd058a1e08e1e37 WatchSource:0}: Error finding container c60e696764b8428326b679d4297dd4826d3cb41f248ad3032cd058a1e08e1e37: Status 404 returned error can't find the container with id c60e696764b8428326b679d4297dd4826d3cb41f248ad3032cd058a1e08e1e37
Oct 30 00:18:31.976991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.976962    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:18:31.977076 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.977039    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:18:31.977076 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:31.977069    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:18:32.000922 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:32.000899    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"]
Oct 30 00:18:32.008263 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:18:32.008234    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d7f0fa_9352_4408_b1a1_cc80af286710.slice/crio-2f2fe6e6fcabefc97277d9f08bbd73d71223a90763faa0f5014776809bea248b WatchSource:0}: Error finding container 2f2fe6e6fcabefc97277d9f08bbd73d71223a90763faa0f5014776809bea248b: Status 404 returned error can't find the container with id 2f2fe6e6fcabefc97277d9f08bbd73d71223a90763faa0f5014776809bea248b
Oct 30 00:18:32.010817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:32.010788    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:18:32.010882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:32.010856    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:18:32.010918 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:32.010885    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:18:32.572078 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:32.571984    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx" event={"ID":"42d7f0fa-9352-4408-b1a1-cc80af286710","Type":"ContainerStarted","Data":"cfe8b748064f61fba58bd2514ba5c22491f9aa5cb715fd5ed49dd90975e18d56"}
Oct 30 00:18:32.572078 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:32.572064    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx" event={"ID":"42d7f0fa-9352-4408-b1a1-cc80af286710","Type":"ContainerStarted","Data":"2f2fe6e6fcabefc97277d9f08bbd73d71223a90763faa0f5014776809bea248b"}
Oct 30 00:18:32.573895 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:32.573865    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk" event={"ID":"fe79b3ad-1c24-4dde-bc30-23a4264e35f9","Type":"ContainerStarted","Data":"d3ebaebd640f60d2f2c002670b7ec7d7c1b21f1e2d9daa07951a29173106bab5"}
Oct 30 00:18:32.574017 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:32.573919    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk" event={"ID":"fe79b3ad-1c24-4dde-bc30-23a4264e35f9","Type":"ContainerStarted","Data":"c60e696764b8428326b679d4297dd4826d3cb41f248ad3032cd058a1e08e1e37"}
Oct 30 00:18:32.616318 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:32.616264    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx" podStartSLOduration=1.6162502399999998 podStartE2EDuration="1.61625024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:32.61457346 +0000 UTC m=+822.139867362" watchObservedRunningTime="2026-10-30 00:18:32.61625024 +0000 UTC m=+822.141544115"
Oct 30 00:18:32.650931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:32.650896    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:32.660810 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:32.660714    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk" podStartSLOduration=1.6606975400000001 podStartE2EDuration="1.66069754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:32.659053832 +0000 UTC m=+822.184347712" watchObservedRunningTime="2026-10-30 00:18:32.66069754 +0000 UTC m=+822.185991416"
Oct 30 00:18:32.676045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:32.675926    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:32.989213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:32.988257    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"]
Oct 30 00:18:33.668051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:33.657870    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:18:33.668051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:33.657948    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk" podUID="fe79b3ad-1c24-4dde-bc30-23a4264e35f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:18:33.697327 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:33.695500    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:18:33.697327 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:33.695569    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx" podUID="42d7f0fa-9352-4408-b1a1-cc80af286710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:18:33.920716 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:18:33.920193    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:18:34.664044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:34.652961    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:18:34.664044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:34.653015    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk" podUID="fe79b3ad-1c24-4dde-bc30-23a4264e35f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:18:34.677431 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:34.677314    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:18:34.677431 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:34.677379    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx" podUID="42d7f0fa-9352-4408-b1a1-cc80af286710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:18:35.673371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:35.670125    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:18:35.673371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:35.670183    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk" podUID="fe79b3ad-1c24-4dde-bc30-23a4264e35f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:18:35.684100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:35.683131    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:18:35.684100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:35.683186    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx" podUID="42d7f0fa-9352-4408-b1a1-cc80af286710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:18:36.660050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:36.659346    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:18:36.660050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:36.659406    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk" podUID="fe79b3ad-1c24-4dde-bc30-23a4264e35f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:18:36.681056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:36.680479    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:18:36.681056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:36.680540    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx" podUID="42d7f0fa-9352-4408-b1a1-cc80af286710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:18:37.116243 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.116201    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"]
Oct 30 00:18:37.209703 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.209668    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"]
Oct 30 00:18:37.209931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.209813    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.417449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.417368    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.417449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.417405    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.417449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.417431    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbk99\" (UniqueName: \"kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-kube-api-access-hbk99\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.417716 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.417485    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.417716 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.417566    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.417716 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.417701    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.417872 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.417765    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.417872 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.417791    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0298952c-f699-4f6b-b32a-f35db6d0e113-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.417872 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.417828    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.519168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.519132    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.519366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.519244    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.519366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.519287    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.519366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.519306    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0298952c-f699-4f6b-b32a-f35db6d0e113-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.519545 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.519507    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.519631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.519612    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.519691 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.519653    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.519691 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.519658    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.519778 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.519708    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbk99\" (UniqueName: \"kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-kube-api-access-hbk99\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.519778 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.519770    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.519945 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.519919    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.520040 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.519994    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.520163 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.520145    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.520206 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.520181    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0298952c-f699-4f6b-b32a-f35db6d0e113-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.525338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.525290    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.531570 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.531541    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.545771 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.545725    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbk99\" (UniqueName: \"kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-kube-api-access-hbk99\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.546412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.546390    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.595748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.595462    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:37.683107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.672575    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:18:37.683107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.672634    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk" podUID="fe79b3ad-1c24-4dde-bc30-23a4264e35f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:18:37.683107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.676341    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:18:37.683107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.676409    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx" podUID="42d7f0fa-9352-4408-b1a1-cc80af286710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:18:37.987215 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.986256    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"]
Oct 30 00:18:37.995078 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.995007    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:18:37.995208 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.995130    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:18:37.995208 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:37.995173    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:18:38.620094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:38.619262    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5" event={"ID":"0298952c-f699-4f6b-b32a-f35db6d0e113","Type":"ContainerStarted","Data":"f53d85cfd3fd9df994d8416d38d9143eae10b778b1ec1a62cc6ff849233c735e"}
Oct 30 00:18:38.620094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:38.619316    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5" event={"ID":"0298952c-f699-4f6b-b32a-f35db6d0e113","Type":"ContainerStarted","Data":"ad313330a4e12e8e4acd731bffcb08d72118a578ec88af7c34086f23fa649a27"}
Oct 30 00:18:38.653056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:38.651215    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:18:38.653056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:38.651278    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk" podUID="fe79b3ad-1c24-4dde-bc30-23a4264e35f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:18:38.663615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:38.663549    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5" podStartSLOduration=1.6635299300000002 podStartE2EDuration="1.66352993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:38.661981129 +0000 UTC m=+828.187275045" watchObservedRunningTime="2026-10-30 00:18:38.66352993 +0000 UTC m=+828.188823810"
Oct 30 00:18:38.676183 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:38.676152    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:18:38.676400 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:38.676207    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx" podUID="42d7f0fa-9352-4408-b1a1-cc80af286710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:18:39.596383 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:39.596334    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:39.598709 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:39.598676    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:18:39.598843 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:39.598762    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5" podUID="0298952c-f699-4f6b-b32a-f35db6d0e113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:18:40.081047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:40.079149    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:40.081047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:40.080644    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:40.081618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:40.081558    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:18:40.087044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:40.083170    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:40.087044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:40.084749    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:40.087044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:40.086389    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:18:40.596112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:40.596072    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:18:40.596302 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:40.596135    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5" podUID="0298952c-f699-4f6b-b32a-f35db6d0e113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:18:41.596386 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:41.596342    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:18:41.596782 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:41.596414    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5" podUID="0298952c-f699-4f6b-b32a-f35db6d0e113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:18:42.596979 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:42.596943    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:18:42.597419 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:42.596994    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5" podUID="0298952c-f699-4f6b-b32a-f35db6d0e113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:18:43.596306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:43.596268    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:18:43.596468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:43.596339    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5" podUID="0298952c-f699-4f6b-b32a-f35db6d0e113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:18:44.597145 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:44.597104    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:18:44.597575 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:44.597175    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5" podUID="0298952c-f699-4f6b-b32a-f35db6d0e113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:18:45.640103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:45.640064    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:45.641805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:45.641777    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:45.642271 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:45.642252    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:18:48.125350 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.125304    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg" podUID="4837c61b-790f-4bc8-ae92-56365684f5d4" containerName="main" containerID="cri-o://41b939d0dda494717e33234098d9b46d525dbd09c93ce8ac5404b0378a68bbc3" gracePeriod=45
Oct 30 00:18:48.419769 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.419741    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:18:48.441988 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.441961    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4837c61b-790f-4bc8-ae92-56365684f5d4-kube-api-access-xtxhh\" (UniqueName: \"kubernetes.io/projected/4837c61b-790f-4bc8-ae92-56365684f5d4-kube-api-access-xtxhh\") pod \"4837c61b-790f-4bc8-ae92-56365684f5d4\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") "
Oct 30 00:18:48.442146 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.442019    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-model-cache\") pod \"4837c61b-790f-4bc8-ae92-56365684f5d4\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") "
Oct 30 00:18:48.442217 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.442156    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-home\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-home\") pod \"4837c61b-790f-4bc8-ae92-56365684f5d4\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") "
Oct 30 00:18:48.442278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.442229    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-tmp-dir\") pod \"4837c61b-790f-4bc8-ae92-56365684f5d4\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") "
Oct 30 00:18:48.442278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.442261    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4837c61b-790f-4bc8-ae92-56365684f5d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/4837c61b-790f-4bc8-ae92-56365684f5d4-tls-certs\") pod \"4837c61b-790f-4bc8-ae92-56365684f5d4\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") "
Oct 30 00:18:48.442386 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.442339    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-model-cache" pod "4837c61b-790f-4bc8-ae92-56365684f5d4" (UID: "4837c61b-790f-4bc8-ae92-56365684f5d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:18:48.442386 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.442359    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-dshm\") pod \"4837c61b-790f-4bc8-ae92-56365684f5d4\" (UID: \"4837c61b-790f-4bc8-ae92-56365684f5d4\") "
Oct 30 00:18:48.443099 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.442856    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-tmp-dir" pod "4837c61b-790f-4bc8-ae92-56365684f5d4" (UID: "4837c61b-790f-4bc8-ae92-56365684f5d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:18:48.443099 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.443052    4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-model-cache\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:18:48.443099 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.443070    4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-tmp-dir\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:18:48.443445 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.443423    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-home" pod "4837c61b-790f-4bc8-ae92-56365684f5d4" (UID: "4837c61b-790f-4bc8-ae92-56365684f5d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:18:48.445016 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.444982    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-dshm" pod "4837c61b-790f-4bc8-ae92-56365684f5d4" (UID: "4837c61b-790f-4bc8-ae92-56365684f5d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:18:48.445302 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.445266    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4837c61b-790f-4bc8-ae92-56365684f5d4-tls-certs" pod "4837c61b-790f-4bc8-ae92-56365684f5d4" (UID: "4837c61b-790f-4bc8-ae92-56365684f5d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:18:48.445561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.445538    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4837c61b-790f-4bc8-ae92-56365684f5d4-kube-api-access-xtxhh" pod "4837c61b-790f-4bc8-ae92-56365684f5d4" (UID: "4837c61b-790f-4bc8-ae92-56365684f5d4"). InnerVolumeSpecName "kube-api-access-xtxhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:18:48.544496 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.544459    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtxhh\" (UniqueName: \"kubernetes.io/projected/4837c61b-790f-4bc8-ae92-56365684f5d4-kube-api-access-xtxhh\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:18:48.544496 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.544493    4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-home\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:18:48.544496 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.544504    4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4837c61b-790f-4bc8-ae92-56365684f5d4-tls-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:18:48.544496 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.544513    4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4837c61b-790f-4bc8-ae92-56365684f5d4-dshm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:18:48.730942 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.730912    4441 generic.go:356] "Generic (PLEG): container finished" podID="4837c61b-790f-4bc8-ae92-56365684f5d4" containerID="41b939d0dda494717e33234098d9b46d525dbd09c93ce8ac5404b0378a68bbc3" exitCode=0
Oct 30 00:18:48.731153 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.730989    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"
Oct 30 00:18:48.731153 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.731004    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg" event={"ID":"4837c61b-790f-4bc8-ae92-56365684f5d4","Type":"ContainerDied","Data":"41b939d0dda494717e33234098d9b46d525dbd09c93ce8ac5404b0378a68bbc3"}
Oct 30 00:18:48.731153 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.731061    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg" event={"ID":"4837c61b-790f-4bc8-ae92-56365684f5d4","Type":"ContainerDied","Data":"c01fe16f461c3908efa3c886f4d0c087bfd6e7a5415bef1f9becc0c1a66b4ecd"}
Oct 30 00:18:48.731153 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.731080    4441 scope.go:122] "RemoveContainer" containerID="41b939d0dda494717e33234098d9b46d525dbd09c93ce8ac5404b0378a68bbc3"
Oct 30 00:18:48.742586 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.742566    4441 scope.go:122] "RemoveContainer" containerID="41b939d0dda494717e33234098d9b46d525dbd09c93ce8ac5404b0378a68bbc3"
Oct 30 00:18:48.742890 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:18:48.742870    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b939d0dda494717e33234098d9b46d525dbd09c93ce8ac5404b0378a68bbc3\": container with ID starting with 41b939d0dda494717e33234098d9b46d525dbd09c93ce8ac5404b0378a68bbc3 not found: ID does not exist" containerID="41b939d0dda494717e33234098d9b46d525dbd09c93ce8ac5404b0378a68bbc3"
Oct 30 00:18:48.742950 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.742900    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b939d0dda494717e33234098d9b46d525dbd09c93ce8ac5404b0378a68bbc3"} err="failed to get container status \"41b939d0dda494717e33234098d9b46d525dbd09c93ce8ac5404b0378a68bbc3\": rpc error: code = NotFound desc = could not find container \"41b939d0dda494717e33234098d9b46d525dbd09c93ce8ac5404b0378a68bbc3\": container with ID starting with 41b939d0dda494717e33234098d9b46d525dbd09c93ce8ac5404b0378a68bbc3 not found: ID does not exist"
Oct 30 00:18:48.782377 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.782334    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"]
Oct 30 00:18:48.794122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:48.794102    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg"]
Oct 30 00:18:49.313329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:49.313286    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-b-cee02204-kserve-66dc7554db-zs8tg" podUID="4837c61b-790f-4bc8-ae92-56365684f5d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 00:18:50.611258 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:50.611222    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4837c61b-790f-4bc8-ae92-56365684f5d4" path="/var/lib/kubelet/pods/4837c61b-790f-4bc8-ae92-56365684f5d4/volumes"
Oct 30 00:18:52.154239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.154187    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"]
Oct 30 00:18:52.158652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.158626    4441 cpu_manager.go:414] "containerMap: removing container" podUID="4837c61b-790f-4bc8-ae92-56365684f5d4" containerName="main"
Oct 30 00:18:52.158827 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.158810    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4837c61b-790f-4bc8-ae92-56365684f5d4" containerName="main"
Oct 30 00:18:52.159116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.159099    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4837c61b-790f-4bc8-ae92-56365684f5d4" containerName="main"
Oct 30 00:18:52.173421 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.173402    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.217050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.217001    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"]
Oct 30 00:18:52.224052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.220215    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.224442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.224415    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7q27\" (UniqueName: \"kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-kube-api-access-s7q27\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.224687 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.224667    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.230274 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.230247    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.230556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.230525    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.238057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.230749    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.239275 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.239242    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.263341 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.263306    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.263761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.263738    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.415116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.405824    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.415116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.405877    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.415116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.405942    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.415116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.405975    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.415116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.406071    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.415116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.406113    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.415116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.406202    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.415116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.406248    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7q27\" (UniqueName: \"kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-kube-api-access-s7q27\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.415116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.406323    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.427257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.417019    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.457263 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.428564    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.457489 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.428802    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.457607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.429483    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.457706 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.429715    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.581715 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.581659    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:18:52.587184 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.587141    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 30 00:18:52.734102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.693928    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7q27\" (UniqueName: \"kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-kube-api-access-s7q27\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.764047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.752800    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.764047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.753435    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.764047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.761504    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.809505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.809255    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:52.890061 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:52.886933    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:18:53.195424 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:53.194603    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 00:18:53.693059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:53.690463    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:18:53.693059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:53.690978    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:18:53.707590 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:53.706792    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:18:53.887102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:53.840732    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 00:18:53.953649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:53.895235    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:18:53.953649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:53.917615    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:18:53.953649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:53.943082    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:18:54.000148 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:54.000118    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"]
Oct 30 00:18:54.011213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:54.011189    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:18:54.021135 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:54.021080    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:18:54.022379 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:54.021236    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:18:54.022379 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:54.021287    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:18:54.918369 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:54.918330    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c" event={"ID":"e9347cdc-b854-4c2f-b056-e01dd2e24b2c","Type":"ContainerStarted","Data":"e15e1f9275889bbcf74378e41ff25e0d837b167ba472c1d8b505af189d342494"}
Oct 30 00:18:54.918844 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:54.918388    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c" event={"ID":"e9347cdc-b854-4c2f-b056-e01dd2e24b2c","Type":"ContainerStarted","Data":"93c807cd02ffc06ca35ec6e0900e80a3faf93174aa24f9eb298d521a0fcc8235"}
Oct 30 00:18:54.987008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:54.986930    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c" podStartSLOduration=2.986912651 podStartE2EDuration="2.986912651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:54.972846138 +0000 UTC m=+844.498140008" watchObservedRunningTime="2026-10-30 00:18:54.986912651 +0000 UTC m=+844.512206529"
Oct 30 00:18:55.809961 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:55.809930    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:18:55.811864 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:55.811837    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:18:55.812012 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:55.811892    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c" podUID="e9347cdc-b854-4c2f-b056-e01dd2e24b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 00:18:56.810258 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:56.810224    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:18:56.810757 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:56.810283    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c" podUID="e9347cdc-b854-4c2f-b056-e01dd2e24b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 00:18:57.810565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:57.810536    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:18:57.810965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:57.810592    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c" podUID="e9347cdc-b854-4c2f-b056-e01dd2e24b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 00:18:58.810270 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:58.810238    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:18:58.810428 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:58.810289    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c" podUID="e9347cdc-b854-4c2f-b056-e01dd2e24b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 00:18:59.810040 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:59.809999    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:18:59.810429 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:18:59.810086    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c" podUID="e9347cdc-b854-4c2f-b056-e01dd2e24b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 00:19:00.810201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:00.810168    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:19:00.810582 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:00.810228    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c" podUID="e9347cdc-b854-4c2f-b056-e01dd2e24b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 00:19:01.963329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:01.963295    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:19:01.963747 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:01.963679    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:19:01.964534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:01.964507    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:19:03.900385 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:19:03.900343    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:19:04.154690 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.154577    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"]
Oct 30 00:19:04.159795 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.159773    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.195701 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.195661    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"]
Oct 30 00:19:04.264433 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.264406    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.264578 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.264487    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.264682 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.264653    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48vf\" (UniqueName: \"kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-kube-api-access-w48vf\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.264747 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.264702    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.264813 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.264787    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.264884 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.264868    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.264928 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.264918    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.264972 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.264960    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.265034 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.265009    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.366168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.366139    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.366337 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.366225    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.366337 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.366272    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.366449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.366339    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48vf\" (UniqueName: \"kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-kube-api-access-w48vf\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.366449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.366358    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.366449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.366375    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.366449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.366426    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.366665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.366469    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.366665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.366514    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.366934 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.366879    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.367080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.366921    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.367157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.367126    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.367218 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.367149    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.367218 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.367183    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.368741 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.368720    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.369215 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.369198    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.391113 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.391087    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.391224 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.391210    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48vf\" (UniqueName: \"kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-kube-api-access-w48vf\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.509423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.509395    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:04.811549 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.811270    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"]
Oct 30 00:19:04.819162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.817011    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.842223 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.841970    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-322d20\"/\"distinct-2-w2-322d20-kserve-self-signed-certs\""
Oct 30 00:19:04.846528 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.846252    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"]
Oct 30 00:19:04.850247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.850221    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"]
Oct 30 00:19:04.851577 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.851543    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:04.851782 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.851733    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:04.851870 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.851813    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:04.874037 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.873330    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-home\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.874037 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.873382    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-tmp-dir\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.874037 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.873472    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7dlp\" (UniqueName: \"kubernetes.io/projected/58d43eef-6285-4674-919c-20246cf8c597-kube-api-access-f7dlp\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.874037 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.873497    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-model-cache\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.874037 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.873598    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-dshm\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.874037 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.873631    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d43eef-6285-4674-919c-20246cf8c597-tls-certs\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.975342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.975307    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7dlp\" (UniqueName: \"kubernetes.io/projected/58d43eef-6285-4674-919c-20246cf8c597-kube-api-access-f7dlp\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.975806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.975348    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-model-cache\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.975806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.975477    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-dshm\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.975806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.975523    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d43eef-6285-4674-919c-20246cf8c597-tls-certs\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.975806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.975640    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-home\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.975806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.975679    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-tmp-dir\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.976214 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.975825    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-model-cache\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.976214 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.976061    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-tmp-dir\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.976214 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.976110    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-home\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.977785 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.977766    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-dshm\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.978486 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.978471    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d43eef-6285-4674-919c-20246cf8c597-tls-certs\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:04.998399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:04.998372    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7dlp\" (UniqueName: \"kubernetes.io/projected/58d43eef-6285-4674-919c-20246cf8c597-kube-api-access-f7dlp\") pod \"distinct-2-w2-322d20-kserve-657769d5b5-fd7hg\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:05.035049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:05.033338    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf" event={"ID":"c81d80c1-da80-4408-8cdf-4bd4acbbfc84","Type":"ContainerStarted","Data":"3ae796120c195ec36f3322c2297d24acd5399dceb59d2745ebcda693793ffad3"}
Oct 30 00:19:05.035049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:05.033408    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf" event={"ID":"c81d80c1-da80-4408-8cdf-4bd4acbbfc84","Type":"ContainerStarted","Data":"11852715374fb701d40e57d5127c3ab284fb14628178549b87c1ccd826c63a9c"}
Oct 30 00:19:05.083744 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:05.082468    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf" podStartSLOduration=1.082446117 podStartE2EDuration="1.082446117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:05.078379757 +0000 UTC m=+854.603673638" watchObservedRunningTime="2026-10-30 00:19:05.082446117 +0000 UTC m=+854.607739995"
Oct 30 00:19:05.149461 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:05.149426    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:05.463095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:05.462661    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"]
Oct 30 00:19:05.509889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:05.509860    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:05.511995 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:05.511967    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:19:05.512111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:05.512052    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf" podUID="c81d80c1-da80-4408-8cdf-4bd4acbbfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:19:06.057124 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:06.057087    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" event={"ID":"58d43eef-6285-4674-919c-20246cf8c597","Type":"ContainerStarted","Data":"84deb3e5877df903fb4f1a91f39e8ab80f53baf9258ec75b7a46aaef1090b5e1"}
Oct 30 00:19:06.057597 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:06.057138    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" event={"ID":"58d43eef-6285-4674-919c-20246cf8c597","Type":"ContainerStarted","Data":"a12cce445b23f97db85688c265dc7248a6ce32de145dd4494e356d94e75c0a75"}
Oct 30 00:19:06.099542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:06.099490    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podStartSLOduration=2.099477423 podStartE2EDuration="2.099477423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:06.098771558 +0000 UTC m=+855.624065440" watchObservedRunningTime="2026-10-30 00:19:06.099477423 +0000 UTC m=+855.624771300"
Oct 30 00:19:06.150050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:06.149603    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:06.510112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:06.510078    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:19:06.510317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:06.510133    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf" podUID="c81d80c1-da80-4408-8cdf-4bd4acbbfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:19:07.510696 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:07.510659    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:19:07.511147 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:07.510721    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf" podUID="c81d80c1-da80-4408-8cdf-4bd4acbbfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:19:08.510238 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:08.510185    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:19:08.510412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:08.510257    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf" podUID="c81d80c1-da80-4408-8cdf-4bd4acbbfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:19:09.510533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:09.510498    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:19:09.510913 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:09.510561    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf" podUID="c81d80c1-da80-4408-8cdf-4bd4acbbfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:19:10.510244 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:10.510208    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:19:10.510433 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:10.510265    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf" podUID="c81d80c1-da80-4408-8cdf-4bd4acbbfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:19:11.510639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:11.510601    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:19:11.511209 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:11.510690    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf" podUID="c81d80c1-da80-4408-8cdf-4bd4acbbfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:19:12.514594 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:12.514560    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:12.514989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:12.514826    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:12.515494 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:12.515476    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:19:15.149881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:15.149849    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:15.167736 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:15.167713    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:16.139777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:16.139744    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:18.075405 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:18.075365    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:19:20.431165 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:20.431117    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:20.680760 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:20.680706    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:19:20.829163 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:20.829119    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:21.081217 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:21.081108    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 00:19:21.095167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:21.095116    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:21.116375 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:21.116320    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:21.211275 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:21.211232    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:21.436324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:21.436197    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:21.439000 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:21.438979    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:19:21.732107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:21.727687    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:21.783404 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:21.783368    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:19:21.892051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:21.844065    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:21.892051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:21.845410    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:19:22.082344 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.081659    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:19:22.140559 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.140520    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:22.141068 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.141040    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:19:22.147174 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.147150    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:19:22.260204 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.254014    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:22.260204 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.256257    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:19:22.289108 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.284920    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:22.293144 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.293101    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:19:22.772051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.763251    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:19:22.786213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.786169    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" podUID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:22.821283 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.804174    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 00:19:22.915224 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.895478    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 00:19:23.040120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.989698    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 30 00:19:23.040120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:22.990952    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:19:23.139087 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.110799    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:19:23.151069 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.143112    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:23.151538 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.151503    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:23.300699 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.300579    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 00:19:23.397397 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.326388    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:23.411441 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.397609    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:23.951802 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:19:23.951768    4441 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.345s"
Oct 30 00:19:23.952458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.952437    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:19:23.952623 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.952608    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"]
Oct 30 00:19:23.970184 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.970163    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"]
Oct 30 00:19:23.970481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.970468    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:23.970618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.970609    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:19:23.973407 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.970404    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:23.989892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.989188    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"distinct-w0-7505bd-kserve-self-signed-certs\""
Oct 30 00:19:23.990734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.990720    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:19:23.990987 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.990969    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:19:24.085179 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:23.998630    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:24.146669 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.146632    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-home\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.146981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.146956    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-dshm\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.147173 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.147153    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tmp-dir\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.148109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.147361    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-model-cache\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.157842 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.157815    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfz9m\" (UniqueName: \"kubernetes.io/projected/7846274a-6b3c-41c5-ab26-45d2d75bc34f-kube-api-access-cfz9m\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.164105 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.160013    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tls-certs\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.213724 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.212009    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:24.214082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.214067    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:19:24.214705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.214684    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:19:24.214956 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.214945    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:19:24.262614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.262584    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:19:24.280854 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.280822    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-model-cache\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.281382 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.281353    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfz9m\" (UniqueName: \"kubernetes.io/projected/7846274a-6b3c-41c5-ab26-45d2d75bc34f-kube-api-access-cfz9m\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.281539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.281522    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tls-certs\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.281793 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.281773    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-home\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.281975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.281961    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-dshm\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.282111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.282096    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tmp-dir\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.287744 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.286901    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tmp-dir\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.288496 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.288476    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-model-cache\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.292548 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.292509    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-home\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.292972 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.292954    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:19:24.299476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.297903    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tls-certs\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.308566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.308543    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-dshm\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.383795 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.383763    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfz9m\" (UniqueName: \"kubernetes.io/projected/7846274a-6b3c-41c5-ab26-45d2d75bc34f-kube-api-access-cfz9m\") pod \"distinct-w0-7505bd-kserve-7d77698984-46vks\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.449563 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.449253    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:24.788820 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:24.788787    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"]
Oct 30 00:19:24.803951 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:19:24.803919    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7846274a_6b3c_41c5_ab26_45d2d75bc34f.slice/crio-510771b1ee17ae61ca94e3210022e311331df60f58fed89207defc3e75536f7e WatchSource:0}: Error finding container 510771b1ee17ae61ca94e3210022e311331df60f58fed89207defc3e75536f7e: Status 404 returned error can't find the container with id 510771b1ee17ae61ca94e3210022e311331df60f58fed89207defc3e75536f7e
Oct 30 00:19:25.300723 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:25.300687    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" event={"ID":"7846274a-6b3c-41c5-ab26-45d2d75bc34f","Type":"ContainerStarted","Data":"fcc1b88adea96d357c953f6c32483c4dbf58956079bd1c16f6813c3683952aa0"}
Oct 30 00:19:25.301239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:25.300730    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" event={"ID":"7846274a-6b3c-41c5-ab26-45d2d75bc34f","Type":"ContainerStarted","Data":"510771b1ee17ae61ca94e3210022e311331df60f58fed89207defc3e75536f7e"}
Oct 30 00:19:25.350514 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:25.350444    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podStartSLOduration=3.350424194 podStartE2EDuration="3.350424194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:25.34056106 +0000 UTC m=+874.865854950" watchObservedRunningTime="2026-10-30 00:19:25.350424194 +0000 UTC m=+874.875718073"
Oct 30 00:19:25.450253 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:25.450210    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:33.904775 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:19:33.904731    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:19:34.450524 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.450477    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:34.463780 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.463748    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:34.861265 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.861231    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"]
Oct 30 00:19:34.865819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.865795    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:34.894688 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.894661    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"]
Oct 30 00:19:34.923526 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.923492    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6606ae96-4a55-400c-8108-fd38d17c5529-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:34.923954 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.923534    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:34.923954 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.923626    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:34.923954 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.923672    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:34.923954 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.923784    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:34.923954 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.923825    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:34.923954 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.923943    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qdv\" (UniqueName: \"kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-kube-api-access-k6qdv\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:34.924282 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.923988    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:34.924282 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:34.924145    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6606ae96-4a55-400c-8108-fd38d17c5529-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.026058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.026004    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6606ae96-4a55-400c-8108-fd38d17c5529-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.026238 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.026067    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.026238 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.026096    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.026238 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.026115    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.026238 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.026148    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.026238 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.026178    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.026493 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.026257    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qdv\" (UniqueName: \"kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-kube-api-access-k6qdv\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.026493 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.026298    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.026493 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.026390    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6606ae96-4a55-400c-8108-fd38d17c5529-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.026683 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.026650    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.026754 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.026729    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.027013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.026992    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.027132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.027044    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.027386 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.027368    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6606ae96-4a55-400c-8108-fd38d17c5529-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.029015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.028994    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.029281 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.029256    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6606ae96-4a55-400c-8108-fd38d17c5529-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.049058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.049015    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.049257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.049240    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qdv\" (UniqueName: \"kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-kube-api-access-k6qdv\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.180790 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.180709    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:35.384807 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.384780    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:35.503537 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.503504    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"]
Oct 30 00:19:35.504049 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:19:35.504000    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6606ae96_4a55_400c_8108_fd38d17c5529.slice/crio-892d92236fe22c1f90a110b46338923ce536d6dcdce7e7d3ce64a17309f288c1 WatchSource:0}: Error finding container 892d92236fe22c1f90a110b46338923ce536d6dcdce7e7d3ce64a17309f288c1: Status 404 returned error can't find the container with id 892d92236fe22c1f90a110b46338923ce536d6dcdce7e7d3ce64a17309f288c1
Oct 30 00:19:35.506539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.506508    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:35.506650 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.506581    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:35.506650 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:35.506612    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:36.382614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:36.382575    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w" event={"ID":"6606ae96-4a55-400c-8108-fd38d17c5529","Type":"ContainerStarted","Data":"88edabdb9186bab7355c2df984c70c0a20d856c172cabecf4a7d8ac6425558f3"}
Oct 30 00:19:36.383117 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:36.382661    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w" event={"ID":"6606ae96-4a55-400c-8108-fd38d17c5529","Type":"ContainerStarted","Data":"892d92236fe22c1f90a110b46338923ce536d6dcdce7e7d3ce64a17309f288c1"}
Oct 30 00:19:37.181308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:37.181269    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:37.183212 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:37.183181    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:19:37.183334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:37.183235    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w" podUID="6606ae96-4a55-400c-8108-fd38d17c5529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 00:19:38.181998 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:38.181962    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:19:38.182403 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:38.182074    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w" podUID="6606ae96-4a55-400c-8108-fd38d17c5529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 00:19:39.181578 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:39.181542    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:19:39.181763 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:39.181620    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w" podUID="6606ae96-4a55-400c-8108-fd38d17c5529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 00:19:40.181498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:40.181464    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:19:40.181944 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:40.181518    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w" podUID="6606ae96-4a55-400c-8108-fd38d17c5529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 00:19:41.181290 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:41.181251    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:19:41.181495 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:41.181325    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w" podUID="6606ae96-4a55-400c-8108-fd38d17c5529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 00:19:42.181908 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:42.181865    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:19:42.182356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:42.181938    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w" podUID="6606ae96-4a55-400c-8108-fd38d17c5529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 00:19:43.185279 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:43.185250    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:43.185633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:43.185614    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:43.186341 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:43.186321    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:19:43.262212 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:43.262149    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w" podStartSLOduration=9.262127166 podStartE2EDuration="9.262127166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:36.450653841 +0000 UTC m=+885.975947722" watchObservedRunningTime="2026-10-30 00:19:43.262127166 +0000 UTC m=+892.787421057"
Oct 30 00:19:44.233500 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.233464    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"]
Oct 30 00:19:44.244467 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.244438    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.277633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.277602    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"]
Oct 30 00:19:44.377752 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.377657    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.377752 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.377697    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.377752 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.377714    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.378062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.377776    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n82rd\" (UniqueName: \"kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-kube-api-access-n82rd\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.378062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.377969    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.378062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.377987    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.378062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.378020    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.378260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.378117    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.378260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.378190    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.478847 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.478818    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.479006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.478887    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.479006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.478932    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.479006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.478962    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.479006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.478990    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.479262 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.479014    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n82rd\" (UniqueName: \"kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-kube-api-access-n82rd\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.479317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.479298    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.479379 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.479330    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.479434 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.479381    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.479434 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.479393    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.479598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.479576    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.479795 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.479768    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.479933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.479907    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.480309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.480284    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.481634 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.481607    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.481754 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.481737    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.501178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.501151    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.501313 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.501295    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n82rd\" (UniqueName: \"kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-kube-api-access-n82rd\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.575239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.575206    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:44.900538 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.900513    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"]
Oct 30 00:19:44.903121 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:19:44.903093    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e3960f_82b3_4e2d_af34_44b4d6b3abec.slice/crio-96caa3dc224b624e9191379161852d946152299b75f7c17cc0f3c4ad8d71d729 WatchSource:0}: Error finding container 96caa3dc224b624e9191379161852d946152299b75f7c17cc0f3c4ad8d71d729: Status 404 returned error can't find the container with id 96caa3dc224b624e9191379161852d946152299b75f7c17cc0f3c4ad8d71d729
Oct 30 00:19:44.906102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.906071    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:44.906171 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.906139    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:44.906213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:44.906172    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:45.445410 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:45.445377    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" event={"ID":"a5e3960f-82b3-4e2d-af34-44b4d6b3abec","Type":"ContainerStarted","Data":"10b4317c4a0f9e5d0367b20f935ffce503213edd90ae0bb6b97e18b7555cf233"}
Oct 30 00:19:45.445821 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:45.445428    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" event={"ID":"a5e3960f-82b3-4e2d-af34-44b4d6b3abec","Type":"ContainerStarted","Data":"96caa3dc224b624e9191379161852d946152299b75f7c17cc0f3c4ad8d71d729"}
Oct 30 00:19:45.494758 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:45.494701    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podStartSLOduration=1.494684791 podStartE2EDuration="1.494684791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:45.492456475 +0000 UTC m=+895.017750378" watchObservedRunningTime="2026-10-30 00:19:45.494684791 +0000 UTC m=+895.019978669"
Oct 30 00:19:45.575660 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:45.575629    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:45.577512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:45.577489    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:19:45.577598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:45.577545    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:19:46.576250 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:46.576208    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:19:46.576654 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:46.576276    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:19:47.586048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:47.576432    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:19:47.586048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:47.576508    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:19:48.579611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:48.579566    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:19:48.579939 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:48.579889    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:19:49.568051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.550275    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"]
Oct 30 00:19:49.568051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.555068    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.591524 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.591045    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:19:49.591524 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.591149    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:19:49.629453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.619990    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.629453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.620048    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.629453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.620140    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.629453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.620190    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7g7r\" (UniqueName: \"kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-kube-api-access-w7g7r\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.629453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.620218    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.629453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.620300    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6544480-6c61-4c34-a724-dc84f756a9e6-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.629453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.620357    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.629453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.620383    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.629453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.620410    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.826059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.716818    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"]
Oct 30 00:19:49.826059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.790104    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.826059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.790151    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.826059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.790182    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.826059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.790315    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.826059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.790340    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.826059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.790440    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.826059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.790493    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7g7r\" (UniqueName: \"kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-kube-api-access-w7g7r\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.826059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.790525    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.826059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.790625    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6544480-6c61-4c34-a724-dc84f756a9e6-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.826059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.791636    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6544480-6c61-4c34-a724-dc84f756a9e6-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.826059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.824832    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.826059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.825136    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.925438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.838789    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.925438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.839070    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:49.967048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:49.952335    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.040648 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.040492    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7g7r\" (UniqueName: \"kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-kube-api-access-w7g7r\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:50.040648 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.040562    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"]
Oct 30 00:19:50.042539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.042485    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:50.069261 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.057198    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:50.072310 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.072239    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:50.080157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.077387    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:50.154065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.105147    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:50.154065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.106566    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:50.154065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.151043    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f48a0a-3901-46b0-adc0-4f447a436671-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.154065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.151107    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.154065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.151234    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.154065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.151278    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.154065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.151370    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.154065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.151407    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f48a0a-3901-46b0-adc0-4f447a436671-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.154065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.151490    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5pzt\" (UniqueName: \"kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-kube-api-access-z5pzt\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.154065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.151539    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.154065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.151614    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.154065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.151945    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:50.217165 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.217115    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"]
Oct 30 00:19:50.270733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.268919    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.270733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.268979    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f48a0a-3901-46b0-adc0-4f447a436671-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.270733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.269077    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5pzt\" (UniqueName: \"kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-kube-api-access-z5pzt\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.270733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.269126    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.270733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.269201    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.270733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.269245    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f48a0a-3901-46b0-adc0-4f447a436671-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.270733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.269300    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.270733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.269469    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.270733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.269532    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.270733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.270015    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.270733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.270295    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.331149 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.295567    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.331149 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.295811    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.331149 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.296434    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f48a0a-3901-46b0-adc0-4f447a436671-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.405948 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.372394    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:50.405948 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.383119    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:19:50.405948 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.385133    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 00:19:50.429063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.421558    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:50.498051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.494408    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.554047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.535763    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f48a0a-3901-46b0-adc0-4f447a436671-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.637144 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.609305    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.658067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.656052    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:19:50.658067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.656131    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:19:50.664980 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.662914    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5pzt\" (UniqueName: \"kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-kube-api-access-z5pzt\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-98kmf\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:50.691060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.684582    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5" podUID="d5deb82b-2372-40b6-b196-3f3f90161ae0" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.17:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:50.788093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.742534    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7c9h5" podUID="d5deb82b-2372-40b6-b196-3f3f90161ae0" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.17:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:50.826336 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:50.826299    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:51.042351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.042311    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:51.118133 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.078703    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:51.118133 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.089070    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 00:19:51.363109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.362241    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 30 00:19:51.363109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.362385    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:19:51.411759 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.411714    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:19:51.430892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.423795    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:19:51.523665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.503498    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:51.766494 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.766448    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:19:51.767181 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.767149    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:19:51.816058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.815664    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:51.821239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.817506    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:19:51.908050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.874582    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:19:51.979052 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:19:51.949201    4441 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.338s"
Oct 30 00:19:51.980089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:51.980064    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:19:52.106102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.101136    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:52.106102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.101194    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:19:52.106102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.101230    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:19:52.106102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.101270    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:52.106102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.101306    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:19:52.106102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.101340    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:19:52.131210 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.097579    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 00:19:52.131210 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.129899    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:19:52.526051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.522232    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:52.566168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.566114    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:52.614102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.566954    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:19:52.797060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.583859    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:19:52.797060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.583892    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:19:52.797060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.583930    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:19:52.797060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.585093    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:19:52.797060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.722153    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:19:52.797060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.722618    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:52.797060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.731830    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:52.850620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.830660    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:19:52.851219 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.831619    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:52.925616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:52.925115    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:19:53.171905 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.157162    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:53.281324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.281278    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 00:19:53.574916 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.541419    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:53.574916 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.569582    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:53.576738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.576627    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:19:53.576738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.576679    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:19:53.645094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.644313    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:53.645094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.644423    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:53.645094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.644485    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:53.670151 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.666784    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:19:53.688457 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.686136    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:53.689253 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.688669    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:53.813719 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.813690    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:19:53.965621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:53.965531    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"]
Oct 30 00:19:54.030522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:54.030485    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 00:19:54.088061 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:54.084660    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:54.235041 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:54.234989    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:54.342263 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:54.342200    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:54.655120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:54.654184    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:19:54.655120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:54.654241    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:19:54.675536 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:54.675493    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:19:54.676093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:54.676059    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:54.709151 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:54.709117    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:54.733881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:54.733813    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:54.734948 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:54.734207    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded"
Oct 30 00:19:54.749253 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:19:54.748580    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6544480_6c61_4c34_a724_dc84f756a9e6.slice/crio-1508ff44b3c62c9abc1581e937283e4233cf89ffe86d31c1bf2d07721608a5e3 WatchSource:0}: Error finding container 1508ff44b3c62c9abc1581e937283e4233cf89ffe86d31c1bf2d07721608a5e3: Status 404 returned error can't find the container with id 1508ff44b3c62c9abc1581e937283e4233cf89ffe86d31c1bf2d07721608a5e3
Oct 30 00:19:54.770067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:54.749463    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:19:55.090474 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:55.075202    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:55.090474 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:55.075695    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:19:55.244105 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:55.215292    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:55.435061 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:55.434544    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 00:19:55.649966 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:55.649930    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:19:55.650261 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:55.650229    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:19:55.650442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:55.650421    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:55.698317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:55.698270    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:19:55.853548 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:55.853437    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 00:19:55.865323 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:55.711347    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:19:55.920361 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:55.920302    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:19:55.979044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:55.972835    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:19:56.000047 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:19:55.995851    4441 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.266s"
Oct 30 00:19:56.049486 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.049446    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" event={"ID":"27f48a0a-3901-46b0-adc0-4f447a436671","Type":"ContainerStarted","Data":"aa9757de0bb19442fac539d98e441673c53e366c5127d5a7a7aa823a4ca16afb"}
Oct 30 00:19:56.080275 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.080225    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 00:19:56.095816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.095777    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:19:56.104986 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.104909    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:19:56.138286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.118992    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:56.138286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.119191    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:56.138286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.119299    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:56.161076 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.161047    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:19:56.201573 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.201543    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:19:56.230944 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.230909    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"]
Oct 30 00:19:56.262629 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.262587    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:56.392806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.392730    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:19:56.406470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.403531    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"]
Oct 30 00:19:56.490387 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.488406    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:19:56.490387 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.488454    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:19:56.490387 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.488484    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:19:56.490387 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.488599    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.490387 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.490076    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"]
Oct 30 00:19:56.490387 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.490395    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:19:56.497911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.497222    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:19:56.581066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.577314    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:19:56.581066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.577362    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:19:56.682565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.680631    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.682565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.680694    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc27936a-64b1-473d-b857-07abf44fdacf-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.682565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.680721    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.682565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.680816    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.682565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.680866    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfqrh\" (UniqueName: \"kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-kube-api-access-wfqrh\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.682565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.680896    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc27936a-64b1-473d-b857-07abf44fdacf-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.682565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.680979    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.682565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.681125    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.682565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.681158    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.804663 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.804621    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc27936a-64b1-473d-b857-07abf44fdacf-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.804850 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.804717    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.804850 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.804793    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.804850 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.804819    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.806429 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.804996    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.806429 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.805059    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc27936a-64b1-473d-b857-07abf44fdacf-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.806429 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.805096    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.806429 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.805167    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.806429 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.805196    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfqrh\" (UniqueName: \"kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-kube-api-access-wfqrh\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.806785 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.806514    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.807487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.807239    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc27936a-64b1-473d-b857-07abf44fdacf-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.808366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.808052    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.808366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.808340    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.810266 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.809267    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.882066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.881985    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc27936a-64b1-473d-b857-07abf44fdacf-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.882926 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.882875    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.889044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.887290    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:56.913897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:56.908131    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfqrh\" (UniqueName: \"kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-kube-api-access-wfqrh\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:57.079049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.076807    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" event={"ID":"d6544480-6c61-4c34-a724-dc84f756a9e6","Type":"ContainerStarted","Data":"c4bc658687983639c999c91890cfdf9bda947381fde4f2ea87fb9230aa440ab6"}
Oct 30 00:19:57.079049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.076862    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" event={"ID":"d6544480-6c61-4c34-a724-dc84f756a9e6","Type":"ContainerStarted","Data":"1508ff44b3c62c9abc1581e937283e4233cf89ffe86d31c1bf2d07721608a5e3"}
Oct 30 00:19:57.081399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.081298    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:19:57.081399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.081354    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" event={"ID":"27f48a0a-3901-46b0-adc0-4f447a436671","Type":"ContainerStarted","Data":"273d59d172fd6f5d2cce2e65d7a1ecc0da601c15e29742629b6399f6143cc1d5"}
Oct 30 00:19:57.083500 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.083215    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:19:57.083500 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.083265    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 00:19:57.120920 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.120861    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" podStartSLOduration=8.120841978 podStartE2EDuration="8.120841978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:57.119857839 +0000 UTC m=+906.645151717" watchObservedRunningTime="2026-10-30 00:19:57.120841978 +0000 UTC m=+906.646135856"
Oct 30 00:19:57.132210 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.131959    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:19:57.207160 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.206596    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" podStartSLOduration=8.206578518 podStartE2EDuration="8.206578518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:57.196497976 +0000 UTC m=+906.721791854" watchObservedRunningTime="2026-10-30 00:19:57.206578518 +0000 UTC m=+906.731872395"
Oct 30 00:19:57.919075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.880774    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"]
Oct 30 00:19:57.919075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.884615    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:19:57.919075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.917857    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:19:57.919075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.917990    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:19:57.982933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.982890    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:57.983188 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.983161    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:57.983411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:57.983388    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:19:58.088048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:58.084322    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:19:58.088048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:58.084387    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 00:19:58.115043 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:58.111988    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" event={"ID":"cc27936a-64b1-473d-b857-07abf44fdacf","Type":"ContainerStarted","Data":"70b4584c44f485d9740bcf6a6320b6151a43672e1cb293273e401b1f328726ed"}
Oct 30 00:19:58.585051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:58.581107    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:19:58.585051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:58.581177    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:58.905112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:58.896177    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:19:58.905112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:58.896244    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:19:59.083050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:59.082479    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:19:59.083050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:59.082543    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 00:19:59.148600 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:59.148560    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" event={"ID":"cc27936a-64b1-473d-b857-07abf44fdacf","Type":"ContainerStarted","Data":"a46bb8f9449d3200c8f1aacf9be7a70a7e6915622eb2b78642d0f917b58c2ab2"}
Oct 30 00:19:59.233629 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:59.230761    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" podStartSLOduration=4.230738811 podStartE2EDuration="4.230738811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:59.224416114 +0000 UTC m=+908.749709989" watchObservedRunningTime="2026-10-30 00:19:59.230738811 +0000 UTC m=+908.756032691"
Oct 30 00:19:59.578325 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:59.578225    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:19:59.625878 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:59.625214    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:19:59.625878 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:59.625286    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:19:59.898100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:59.894866    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:19:59.898100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:19:59.894926    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:20:00.119050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:00.118604    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:20:00.119050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:00.118671    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 00:20:00.152102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:00.141349    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:20:00.152284 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:00.152223    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:20:00.152369 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:00.152281    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:20:00.604056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:00.594494    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:20:00.701050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:00.698166    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:00.847052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:00.846579    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:20:00.899241 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:00.899144    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:20:00.899516 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:00.899485    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:20:01.094201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:01.094135    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 00:20:01.143066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:01.127100    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 30 00:20:01.143066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:01.127721    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:20:01.143066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:01.127764    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 00:20:01.143548 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:01.143517    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:20:01.143746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:01.143711    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:20:01.643051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:01.642319    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:20:01.705099 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:01.705048    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:01.706522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:01.706496    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:20:01.987107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:01.972460    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:20:01.987107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:01.972543    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:20:02.082481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:02.082450    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:20:02.082754 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:02.082727    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 00:20:02.121073 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:02.116102    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:20:02.219405 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:02.148712    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:20:02.219760 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:02.219645    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:20:02.219909 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:02.164399    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:02.328163 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:02.328079    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:20:02.583172 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:02.583071    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:02.886889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:02.886796    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:20:02.887480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:02.887445    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:20:03.081616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.081072    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:20:03.081616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.081145    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 00:20:03.190268 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.190171    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:20:03.190545 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.190500    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:20:03.419120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.411959    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"]
Oct 30 00:20:03.442942 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.442864    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"
Oct 30 00:20:03.468061 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.465196    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"]
Oct 30 00:20:03.558341 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.558309    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3fae6d\""
Oct 30 00:20:03.584212 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.584182    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3fae6d-dockercfg-kkdp6\""
Oct 30 00:20:03.600279 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.600238    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13255825-7146-4789-9262-3cb401f87bd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f\" (UID: \"13255825-7146-4789-9262-3cb401f87bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"
Oct 30 00:20:03.611058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.600865    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmvjv\" (UniqueName: \"kubernetes.io/projected/13255825-7146-4789-9262-3cb401f87bd3-kube-api-access-xmvjv\") pod \"payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f\" (UID: \"13255825-7146-4789-9262-3cb401f87bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"
Oct 30 00:20:03.692247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.690947    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"]
Oct 30 00:20:03.714688 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.714594    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13255825-7146-4789-9262-3cb401f87bd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f\" (UID: \"13255825-7146-4789-9262-3cb401f87bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"
Oct 30 00:20:03.716056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.715853    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13255825-7146-4789-9262-3cb401f87bd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f\" (UID: \"13255825-7146-4789-9262-3cb401f87bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"
Oct 30 00:20:03.732133 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.727821    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"
Oct 30 00:20:03.769971 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.769945    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmvjv\" (UniqueName: \"kubernetes.io/projected/13255825-7146-4789-9262-3cb401f87bd3-kube-api-access-xmvjv\") pod \"payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f\" (UID: \"13255825-7146-4789-9262-3cb401f87bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"
Oct 30 00:20:03.802535 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.777630    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"]
Oct 30 00:20:03.841856 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.841789    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmvjv\" (UniqueName: \"kubernetes.io/projected/13255825-7146-4789-9262-3cb401f87bd3-kube-api-access-xmvjv\") pod \"payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f\" (UID: \"13255825-7146-4789-9262-3cb401f87bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"
Oct 30 00:20:03.885087 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.885052    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:20:03.885348 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.885317    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:20:03.908075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.907002    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djxg8\" (UniqueName: \"kubernetes.io/projected/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-kube-api-access-djxg8\") pod \"payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2\" (UID: \"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01\") " pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"
Oct 30 00:20:03.908075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:03.907211    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2\" (UID: \"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01\") " pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"
Oct 30 00:20:03.983970 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:20:03.982800    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:20:04.008931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.008896    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djxg8\" (UniqueName: \"kubernetes.io/projected/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-kube-api-access-djxg8\") pod \"payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2\" (UID: \"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01\") " pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"
Oct 30 00:20:04.009131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.009106    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2\" (UID: \"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01\") " pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"
Oct 30 00:20:04.010386 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.010335    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2\" (UID: \"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01\") " pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"
Oct 30 00:20:04.100060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.087228    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:20:04.100060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.087292    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 00:20:04.100060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.089137    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djxg8\" (UniqueName: \"kubernetes.io/projected/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-kube-api-access-djxg8\") pod \"payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2\" (UID: \"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01\") " pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"
Oct 30 00:20:04.116821 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.116559    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"
Oct 30 00:20:04.138046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.134292    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"
Oct 30 00:20:04.164407 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.138317    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:20:04.164407 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.138367    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:20:04.658141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.653724    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"]
Oct 30 00:20:04.684951 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.684904    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"]
Oct 30 00:20:04.695920 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:20:04.693873    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a58ad1_9196_49a3_a61a_a3e7ebb5cc01.slice/crio-9a8008f8e919a105ead1b0aa3c4ece392876701b5fb5b0509e25b2edaea3b0e6 WatchSource:0}: Error finding container 9a8008f8e919a105ead1b0aa3c4ece392876701b5fb5b0509e25b2edaea3b0e6: Status 404 returned error can't find the container with id 9a8008f8e919a105ead1b0aa3c4ece392876701b5fb5b0509e25b2edaea3b0e6
Oct 30 00:20:04.814208 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.814170    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2" event={"ID":"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01","Type":"ContainerStarted","Data":"9a8008f8e919a105ead1b0aa3c4ece392876701b5fb5b0509e25b2edaea3b0e6"}
Oct 30 00:20:04.815297 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.815271    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f" event={"ID":"13255825-7146-4789-9262-3cb401f87bd3","Type":"ContainerStarted","Data":"048a30e97261bf9f9ee66eaf390d8a0ffcd88854d4f59b254c8c0d358d06d35f"}
Oct 30 00:20:04.891308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.891175    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:20:04.891308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:04.891234    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:20:05.084461 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:05.083271    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:20:05.084461 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:05.084072    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:20:05.084461 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:05.084117    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 00:20:05.134751 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:05.134365    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:20:05.134751 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:05.134419    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:20:05.842050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:05.841650    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2" event={"ID":"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01","Type":"ContainerStarted","Data":"f1cc0158c5b1bc07947ac340c2f71ae644719062e37873ed86dfccd204ed0993"}
Oct 30 00:20:05.847048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:05.842821    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"
Oct 30 00:20:05.847048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:05.845395    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f" event={"ID":"13255825-7146-4789-9262-3cb401f87bd3","Type":"ContainerStarted","Data":"6ab4630f8fc8445453b910ba6d03a9cdd76ede98e7c852a8ed901822d057b00a"}
Oct 30 00:20:05.860128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:05.855081    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"
Oct 30 00:20:05.881301 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:05.881270    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:20:05.887533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:05.887507    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:20:05.888335 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:05.888112    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2" podStartSLOduration=2.888093793 podStartE2EDuration="2.888093793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:05.884942859 +0000 UTC m=+915.410236737" watchObservedRunningTime="2026-10-30 00:20:05.888093793 +0000 UTC m=+915.413387669"
Oct 30 00:20:05.999901 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:05.999839    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f" podStartSLOduration=2.99981081 podStartE2EDuration="2.99981081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:05.942924046 +0000 UTC m=+915.468217927" watchObservedRunningTime="2026-10-30 00:20:05.99981081 +0000 UTC m=+915.525104687"
Oct 30 00:20:06.109440 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:06.109334    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:20:06.113222 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:06.113197    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:20:06.132920 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:06.132890    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:20:06.133155 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:06.132941    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:20:06.854781 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:06.854750    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:20:07.425601 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:07.425566    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:20:07.426072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:07.425981    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:20:07.426791 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:07.426770    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:20:08.946739 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:08.946701    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"]
Oct 30 00:20:08.965433 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:08.965397    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:08.977070 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:08.977044    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"]
Oct 30 00:20:09.086591 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.086556    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mrv6\" (UniqueName: \"kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-kube-api-access-9mrv6\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.086781 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.086693    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.086845 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.086787    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.086913 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.086893    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.086962 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.086934    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.087014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.086960    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.087084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.087046    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.087131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.087088    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.087171 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.087132    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.188941 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.188899    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mrv6\" (UniqueName: \"kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-kube-api-access-9mrv6\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.189133 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.188982    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.189133 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.189064    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.189133 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.189119    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.189300 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.189140    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.189300 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.189155    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.189300 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.189237    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.189441 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.189310    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.189441 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.189366    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.189568 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.189536    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.189692 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.189666    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.189835 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.189811    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.189905 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.189815    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.189905 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.189862    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.191641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.191619    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.191723 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.191704    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.211007 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.210956    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.211465 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.211444    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mrv6\" (UniqueName: \"kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-kube-api-access-9mrv6\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.283929 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.283903    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:09.633225 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:20:09.633197    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c8a2b4_c0aa_41f2_a0af_80de573b6fb5.slice/crio-5f9d1400f31b775a8b94c59b0a3d742be6cdbc5f182424f7ceb5aea5d59849d5 WatchSource:0}: Error finding container 5f9d1400f31b775a8b94c59b0a3d742be6cdbc5f182424f7ceb5aea5d59849d5: Status 404 returned error can't find the container with id 5f9d1400f31b775a8b94c59b0a3d742be6cdbc5f182424f7ceb5aea5d59849d5
Oct 30 00:20:09.633577 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.633552    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"]
Oct 30 00:20:09.636196 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.636165    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:09.636285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.636226    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:09.636285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.636258    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:09.741083 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.741051    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"]
Oct 30 00:20:09.746166 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.746142    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:09.787759 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.785081    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0847e8c0-openshift-default-dockercfg-n77t7\""
Oct 30 00:20:09.787914 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.787799    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"]
Oct 30 00:20:09.876361 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.876318    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" event={"ID":"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5","Type":"ContainerStarted","Data":"07eff54e2c6dad24f82f32fb4d687818072a77095b7d55cf47697ea209de4875"}
Oct 30 00:20:09.876545 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.876375    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" event={"ID":"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5","Type":"ContainerStarted","Data":"5f9d1400f31b775a8b94c59b0a3d742be6cdbc5f182424f7ceb5aea5d59849d5"}
Oct 30 00:20:09.901851 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.901684    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-podinfo\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:09.901851 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.901826    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-data\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:09.902121 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.902091    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-socket\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:09.903584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.902238    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istiod-ca-cert\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:09.903584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.902378    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-credential-socket\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:09.903584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.902571    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8gbt\" (UniqueName: \"kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-kube-api-access-f8gbt\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:09.903584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.902725    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-token\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:09.903584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.902802    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-certs\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:09.903584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.902883    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-envoy\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:09.970663 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:09.970568    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" podStartSLOduration=1.970546637 podStartE2EDuration="1.970546637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:09.919556627 +0000 UTC m=+919.444850535" watchObservedRunningTime="2026-10-30 00:20:09.970546637 +0000 UTC m=+919.495840515"
Oct 30 00:20:10.007135 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.007094    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istiod-ca-cert\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.008719 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.008690    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-credential-socket\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.010974 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.008959    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8gbt\" (UniqueName: \"kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-kube-api-access-f8gbt\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.011236 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.011216    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-token\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.011376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.011363    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-certs\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.011473 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.011462    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-envoy\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.013370 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.011635    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-podinfo\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.013546 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.013526    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-data\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.013759 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.013724    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-socket\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.014360 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.014322    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-socket\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.014520 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.008585    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istiod-ca-cert\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.014929 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.014892    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-credential-socket\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.016070 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.016039    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-certs\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.017046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.016759    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-data\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.020413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.020123    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-podinfo\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.020413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.020292    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-envoy\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.055507 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.052552    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-token\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.055507 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.053185    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8gbt\" (UniqueName: \"kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-kube-api-access-f8gbt\") pod \"e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") " pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.087039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.086243    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:10.284375 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.284331    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:10.286274 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.286247    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:20:10.286412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.286301    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" podUID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 00:20:10.463088 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.448953    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"]
Oct 30 00:20:10.463088 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.452795    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:10.463088 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.452871    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:10.463088 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.452910    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:10.594125 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.594087    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"]
Oct 30 00:20:10.603461 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.600888    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.603461 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.601473    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"]
Oct 30 00:20:10.675740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.674913    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j86q\" (UniqueName: \"kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-kube-api-access-7j86q\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.675740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.674972    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d355c2f5-010e-409c-81e7-8f1ff07db365-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.675740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.675046    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.675740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.675098    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.675740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.675150    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.675740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.675218    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.675740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.675297    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.675740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.675333    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.675740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.675470    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.776670    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.776735    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.776781    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.778392    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.778487    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.778532    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.778682    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.778817    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j86q\" (UniqueName: \"kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-kube-api-access-7j86q\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.778863    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d355c2f5-010e-409c-81e7-8f1ff07db365-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.779812    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.779835    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d355c2f5-010e-409c-81e7-8f1ff07db365-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.780126    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.780388    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.780738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.780469    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.790096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.782920    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.790096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.783350    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.805634 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.805567    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j86q\" (UniqueName: \"kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-kube-api-access-7j86q\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.813858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.811945    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:10.886095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.886057    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" event={"ID":"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5","Type":"ContainerStarted","Data":"31080c30ea6d0f0fbca5e10172fc2bbdb3f4bddd766d5a78c86fcc66e5b188b9"}
Oct 30 00:20:10.886286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.886124    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" event={"ID":"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5","Type":"ContainerStarted","Data":"1ce6d40a2fc33f6fc2759444d89ee7ee34a9afab1214f5c8bc3b732b718ee149"}
Oct 30 00:20:10.944051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.936805    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" podStartSLOduration=1.936785171 podStartE2EDuration="1.936785171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:10.934317494 +0000 UTC m=+920.459611382" watchObservedRunningTime="2026-10-30 00:20:10.936785171 +0000 UTC m=+920.462079049"
Oct 30 00:20:10.955544 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:10.955340    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:11.129318 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:11.127483    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:11.156309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:11.153158    4441 patch_prober.go:29] interesting pod/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:20:11.156309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:11.153224    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:20:11.285155 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:11.285070    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:20:11.285155 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:11.285124    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" podUID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 00:20:11.311708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:11.311467    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"]
Oct 30 00:20:11.313464 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:20:11.313432    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd355c2f5_010e_409c_81e7_8f1ff07db365.slice/crio-c4c90fa45308973ba2ca475ad09527ea63824a69b2618085b90d2ea80ad5d0ad WatchSource:0}: Error finding container c4c90fa45308973ba2ca475ad09527ea63824a69b2618085b90d2ea80ad5d0ad: Status 404 returned error can't find the container with id c4c90fa45308973ba2ca475ad09527ea63824a69b2618085b90d2ea80ad5d0ad
Oct 30 00:20:11.316870 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:11.316831    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:11.317052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:11.316915    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:11.317052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:11.316956    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:11.894738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:11.894699    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" event={"ID":"d355c2f5-010e-409c-81e7-8f1ff07db365","Type":"ContainerStarted","Data":"94ae8fff20855af9f41573af92274f8ac6f70c94dcdbf230b9dfe37a2288daf4"}
Oct 30 00:20:11.894925 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:11.894764    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" event={"ID":"d355c2f5-010e-409c-81e7-8f1ff07db365","Type":"ContainerStarted","Data":"c4c90fa45308973ba2ca475ad09527ea63824a69b2618085b90d2ea80ad5d0ad"}
Oct 30 00:20:11.956175 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:11.956111    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:12.093453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:12.093426    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:12.146107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:12.141619    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" podStartSLOduration=2.141597799 podStartE2EDuration="2.141597799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:11.954977256 +0000 UTC m=+921.480271134" watchObservedRunningTime="2026-10-30 00:20:12.141597799 +0000 UTC m=+921.666891679"
Oct 30 00:20:12.284580 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:12.284542    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:20:12.284757 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:12.284593    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" podUID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 00:20:12.902974 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:12.902908    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:12.904525 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:12.904498    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:20:12.959241 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:12.958902    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:20:12.959241 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:12.958963    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 00:20:13.286305 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:13.284867    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:20:13.286305 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:13.284940    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" podUID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 00:20:13.965048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:13.964490    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:20:13.965048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:13.964558    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 00:20:14.288065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:14.287334    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:20:14.288065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:14.287386    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" podUID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 00:20:14.979881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:14.969016    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:20:14.979881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:14.969115    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 00:20:15.289989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:15.289303    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:20:15.289989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:15.289362    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" podUID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 00:20:15.988050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:15.982350    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:20:15.988050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:15.982409    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 00:20:16.144111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.131754    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"]
Oct 30 00:20:16.229051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.205534    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"
Oct 30 00:20:16.255111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.249607    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccb4296-5590-42ff-9ea8-870c68993622-plugins-config-volume\") pod \"payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p\" (UID: \"eccb4296-5590-42ff-9ea8-870c68993622\") " pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"
Oct 30 00:20:16.255111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.249657    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnmmw\" (UniqueName: \"kubernetes.io/projected/eccb4296-5590-42ff-9ea8-870c68993622-kube-api-access-qnmmw\") pod \"payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p\" (UID: \"eccb4296-5590-42ff-9ea8-870c68993622\") " pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"
Oct 30 00:20:16.268228 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.255514    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0847e8c0-dockercfg-7g6l5\""
Oct 30 00:20:16.268464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.259157    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0847e8c0\""
Oct 30 00:20:16.296084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.296050    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:20:16.327055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.296650    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" podUID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 00:20:16.328096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.327261    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"]
Oct 30 00:20:16.403947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.363184    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccb4296-5590-42ff-9ea8-870c68993622-plugins-config-volume\") pod \"payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p\" (UID: \"eccb4296-5590-42ff-9ea8-870c68993622\") " pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"
Oct 30 00:20:16.403947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.363233    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnmmw\" (UniqueName: \"kubernetes.io/projected/eccb4296-5590-42ff-9ea8-870c68993622-kube-api-access-qnmmw\") pod \"payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p\" (UID: \"eccb4296-5590-42ff-9ea8-870c68993622\") " pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"
Oct 30 00:20:16.403947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.364900    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccb4296-5590-42ff-9ea8-870c68993622-plugins-config-volume\") pod \"payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p\" (UID: \"eccb4296-5590-42ff-9ea8-870c68993622\") " pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"
Oct 30 00:20:16.415818 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.415785    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnmmw\" (UniqueName: \"kubernetes.io/projected/eccb4296-5590-42ff-9ea8-870c68993622-kube-api-access-qnmmw\") pod \"payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p\" (UID: \"eccb4296-5590-42ff-9ea8-870c68993622\") " pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"
Oct 30 00:20:16.493749 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.493721    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"
Oct 30 00:20:16.680851 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.680794    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"]
Oct 30 00:20:16.789508 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.788726    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.789508 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.789364    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"]
Oct 30 00:20:16.887924 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.887890    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lb2r\" (UniqueName: \"kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-kube-api-access-4lb2r\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.888197 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.888170    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.888384 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.888364    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.888529 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.888512    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4489355f-aad2-4394-bd10-7a78024d6458-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.888840 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.888817    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.889047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.889013    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.889184 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.889166    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.889318 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.889297    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.889445 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.889428    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4489355f-aad2-4394-bd10-7a78024d6458-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.898972 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.898913    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"]
Oct 30 00:20:16.962387 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.962351    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:20:16.962533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.962428    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 00:20:16.990998 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.990969    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.991122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.991011    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.991122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.991082    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.991122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.991114    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4489355f-aad2-4394-bd10-7a78024d6458-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.991255 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.991238    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lb2r\" (UniqueName: \"kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-kube-api-access-4lb2r\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.991339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.991322    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.991915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.991385    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.991915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.991439    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4489355f-aad2-4394-bd10-7a78024d6458-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.991915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.991619    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.991915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.991640    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.992247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.992223    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.992481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.992455    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4489355f-aad2-4394-bd10-7a78024d6458-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.992629 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.992606    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.992693 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.992662    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.993814 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.993794    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:16.994195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:16.994180    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4489355f-aad2-4394-bd10-7a78024d6458-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:17.013658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.013428    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:17.013658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.013623    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lb2r\" (UniqueName: \"kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-kube-api-access-4lb2r\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:17.067103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.066491    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p" event={"ID":"eccb4296-5590-42ff-9ea8-870c68993622","Type":"ContainerStarted","Data":"db3d62d66eb15206939106b53205411498d79790ecaeb70f5eae848f17fd8517"}
Oct 30 00:20:17.067103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.066546    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p" event={"ID":"eccb4296-5590-42ff-9ea8-870c68993622","Type":"ContainerStarted","Data":"54a0dfd568b6f7ec0aa9010203f957b937e4f74d20f524c37b61696e386c51ad"}
Oct 30 00:20:17.067103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.066739    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"
Oct 30 00:20:17.109908 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.109873    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:17.111562 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.111521    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p" podStartSLOduration=2.111508798 podStartE2EDuration="2.111508798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:17.107563162 +0000 UTC m=+926.632857063" watchObservedRunningTime="2026-10-30 00:20:17.111508798 +0000 UTC m=+926.636802675"
Oct 30 00:20:17.284838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.284368    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:20:17.284838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.284436    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" podUID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 00:20:17.580129 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.579999    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"]
Oct 30 00:20:17.587572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.587531    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:17.587686 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.587597    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:17.587686 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.587625    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:17.889078 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.888981    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"
Oct 30 00:20:17.897694 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.896149    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"
Oct 30 00:20:17.974047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.966584    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:20:17.974047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:17.966640    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 00:20:18.120071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:18.119855    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw" event={"ID":"4489355f-aad2-4394-bd10-7a78024d6458","Type":"ContainerStarted","Data":"9ec72f9547d83ad4ee92e17fd4e5e961e148b98f9ac2cafc80a6202423eecd11"}
Oct 30 00:20:18.120071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:18.119913    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw" event={"ID":"4489355f-aad2-4394-bd10-7a78024d6458","Type":"ContainerStarted","Data":"872dd15b51cace86314508034b4981efe7a6f512881c213f0dc01b07b8b5ca61"}
Oct 30 00:20:18.191544 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:18.191172    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw" podStartSLOduration=2.191151663 podStartE2EDuration="2.191151663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:18.184070289 +0000 UTC m=+927.709364169" watchObservedRunningTime="2026-10-30 00:20:18.191151663 +0000 UTC m=+927.716445541"
Oct 30 00:20:18.398381 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:18.398345    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:18.399081 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:18.399062    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:18.399977 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:18.399954    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:20:18.960047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:18.956231    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:20:18.960047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:18.956297    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 00:20:19.110981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:19.110949    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:19.112069 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:19.112041    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 00:20:19.112206 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:19.112095    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw" podUID="4489355f-aad2-4394-bd10-7a78024d6458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 00:20:19.961167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:19.961136    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:19.961667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:19.961647    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:19.962524 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:19.962499    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:20:20.115893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:20.111131    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 00:20:20.115893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:20.111191    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw" podUID="4489355f-aad2-4394-bd10-7a78024d6458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 00:20:21.111117 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:21.111083    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 00:20:21.111598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:21.111139    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw" podUID="4489355f-aad2-4394-bd10-7a78024d6458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 00:20:22.110340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:22.110310    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 00:20:22.110509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:22.110363    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw" podUID="4489355f-aad2-4394-bd10-7a78024d6458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 00:20:23.111051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:23.110996    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 00:20:23.111459 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:23.111080    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw" podUID="4489355f-aad2-4394-bd10-7a78024d6458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 00:20:24.110586 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:24.110554    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 00:20:24.110763 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:24.110606    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw" podUID="4489355f-aad2-4394-bd10-7a78024d6458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 00:20:25.141621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:25.141586    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:25.142375 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:25.142348    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:25.144630 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:25.144608    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:20:25.321208 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:25.321170    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:20:28.101152 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:28.101106    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:20:28.162050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:28.150194    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"
Oct 30 00:20:28.537049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:28.534103    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:20:28.738049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:28.712435    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:28.741260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:28.713391    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:29.119048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.118131    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:29.119569 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.119366    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:20:29.231045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.213118    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:29.330793 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.307571    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:20:29.513102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.492603    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:29.574060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.573936    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:29.620957 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.603131    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"]
Oct 30 00:20:29.671049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.635996    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"]
Oct 30 00:20:29.671049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.638913    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:29.671049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.654814    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0847e8c0-serving-cert\""
Oct 30 00:20:29.882014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.881923    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-maas-api-tls\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-x6wxl\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:29.882297 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.882267    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-metrics-certs\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-x6wxl\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:29.882415 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.882322    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402d330d-6644-40e9-9bbe-8775bd0fadc4-openshift-service-ca\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-x6wxl\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:29.882520 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.882497    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drnqj\" (UniqueName: \"kubernetes.io/projected/402d330d-6644-40e9-9bbe-8775bd0fadc4-kube-api-access-drnqj\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-x6wxl\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:29.984210 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.984175    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-metrics-certs\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-x6wxl\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:29.984380 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.984219    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402d330d-6644-40e9-9bbe-8775bd0fadc4-openshift-service-ca\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-x6wxl\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:29.984380 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.984299    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drnqj\" (UniqueName: \"kubernetes.io/projected/402d330d-6644-40e9-9bbe-8775bd0fadc4-kube-api-access-drnqj\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-x6wxl\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:29.984514 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.984488    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-maas-api-tls\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-x6wxl\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:29.985015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.984988    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402d330d-6644-40e9-9bbe-8775bd0fadc4-openshift-service-ca\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-x6wxl\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:29.986739 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.986714    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-metrics-certs\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-x6wxl\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:29.986827 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:29.986771    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-maas-api-tls\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-x6wxl\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:30.006214 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:30.006192    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drnqj\" (UniqueName: \"kubernetes.io/projected/402d330d-6644-40e9-9bbe-8775bd0fadc4-kube-api-access-drnqj\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-x6wxl\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:30.110211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:30.110180    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:30.437328 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:30.437297    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"]
Oct 30 00:20:30.437758 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:20:30.437735    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402d330d_6644_40e9_9bbe_8775bd0fadc4.slice/crio-cba71bfdc100ccbe2d8ad362b04b61da46bdc4d94482afaa91567fb6788fd6dd WatchSource:0}: Error finding container cba71bfdc100ccbe2d8ad362b04b61da46bdc4d94482afaa91567fb6788fd6dd: Status 404 returned error can't find the container with id cba71bfdc100ccbe2d8ad362b04b61da46bdc4d94482afaa91567fb6788fd6dd
Oct 30 00:20:30.440012 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:30.439990    4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:20:30.860047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:30.859710    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl" event={"ID":"402d330d-6644-40e9-9bbe-8775bd0fadc4","Type":"ContainerStarted","Data":"43e63ed303a196c7b5fe2985cdde441f05efae67cfaec34661205711ac7031e1"}
Oct 30 00:20:30.860047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:30.859779    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl" event={"ID":"402d330d-6644-40e9-9bbe-8775bd0fadc4","Type":"ContainerStarted","Data":"cba71bfdc100ccbe2d8ad362b04b61da46bdc4d94482afaa91567fb6788fd6dd"}
Oct 30 00:20:30.861212 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:30.861186    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:30.902595 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:30.902543    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl" podStartSLOduration=1.592104032 podStartE2EDuration="1.902529782s" totalImagesPullingTime="310.42575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:30.900197773 +0000 UTC m=+940.425491652" watchObservedRunningTime="2026-10-30 00:20:30.902529782 +0000 UTC m=+940.427823658"
Oct 30 00:20:33.912077 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:20:33.911962    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:20:37.883612 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:37.883581    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:20:43.762576 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:43.762540    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"]
Oct 30 00:20:43.768362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:43.768020    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"
Oct 30 00:20:43.813055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:43.795275    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"]
Oct 30 00:20:43.813055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:43.795692    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7417dc\""
Oct 30 00:20:43.813055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:43.795867    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7417dc-dockercfg-67vkp\""
Oct 30 00:20:43.849045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:43.814489    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57\" (UID: \"79e0cda7-5bcb-4c13-98cc-92b7a9144b31\") " pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"
Oct 30 00:20:43.864048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:43.849641    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgdrm\" (UniqueName: \"kubernetes.io/projected/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-kube-api-access-vgdrm\") pod \"payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57\" (UID: \"79e0cda7-5bcb-4c13-98cc-92b7a9144b31\") " pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"
Oct 30 00:20:43.966876 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:43.966445    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57\" (UID: \"79e0cda7-5bcb-4c13-98cc-92b7a9144b31\") " pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"
Oct 30 00:20:43.966876 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:43.966549    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdrm\" (UniqueName: \"kubernetes.io/projected/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-kube-api-access-vgdrm\") pod \"payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57\" (UID: \"79e0cda7-5bcb-4c13-98cc-92b7a9144b31\") " pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"
Oct 30 00:20:43.968075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:43.968043    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57\" (UID: \"79e0cda7-5bcb-4c13-98cc-92b7a9144b31\") " pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"
Oct 30 00:20:43.990310 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:43.990282    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdrm\" (UniqueName: \"kubernetes.io/projected/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-kube-api-access-vgdrm\") pod \"payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57\" (UID: \"79e0cda7-5bcb-4c13-98cc-92b7a9144b31\") " pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"
Oct 30 00:20:44.097337 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:44.097242    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"
Oct 30 00:20:44.426421 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:20:44.426387    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e0cda7_5bcb_4c13_98cc_92b7a9144b31.slice/crio-fa2cf6a862ccf71f549e3f25735a4265e14044c1f0c38fee51a4e1c74d399f17 WatchSource:0}: Error finding container fa2cf6a862ccf71f549e3f25735a4265e14044c1f0c38fee51a4e1c74d399f17: Status 404 returned error can't find the container with id fa2cf6a862ccf71f549e3f25735a4265e14044c1f0c38fee51a4e1c74d399f17
Oct 30 00:20:44.426734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:44.426710    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"]
Oct 30 00:20:44.999178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:44.999135    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57" event={"ID":"79e0cda7-5bcb-4c13-98cc-92b7a9144b31","Type":"ContainerStarted","Data":"99201956787ca385626119b03183d7b0a477fda613a39b02022acea4501ee0f2"}
Oct 30 00:20:44.999649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:44.999208    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57" event={"ID":"79e0cda7-5bcb-4c13-98cc-92b7a9144b31","Type":"ContainerStarted","Data":"fa2cf6a862ccf71f549e3f25735a4265e14044c1f0c38fee51a4e1c74d399f17"}
Oct 30 00:20:44.999649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:44.999338    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"
Oct 30 00:20:45.040818 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:45.040757    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57" podStartSLOduration=2.04074221 podStartE2EDuration="2.04074221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:45.037101024 +0000 UTC m=+954.562394901" watchObservedRunningTime="2026-10-30 00:20:45.04074221 +0000 UTC m=+954.566036086"
Oct 30 00:20:46.541003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.540964    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"]
Oct 30 00:20:46.546903 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.546879    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.567682 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.567652    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"]
Oct 30 00:20:46.574853 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.574829    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e2a011-openshift-default-dockercfg-dv5vx\""
Oct 30 00:20:46.727894 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.727854    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-istio-token\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.728249 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.728227    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-data\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.728476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.728440    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xqwj\" (UniqueName: \"kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-kube-api-access-7xqwj\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.728685 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.728668    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-credential-socket\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.728857 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.728841    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-socket\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.729114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.729097    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d7723e-9a7d-4858-8093-4b13593809db-istio-podinfo\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.729392 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.729375    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-envoy\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.729549 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.729536    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-certs\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.731064 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.731044    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d7723e-9a7d-4858-8093-4b13593809db-istiod-ca-cert\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.836491    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d7723e-9a7d-4858-8093-4b13593809db-istio-podinfo\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.836630    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-envoy\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.836663    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-certs\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.836748    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d7723e-9a7d-4858-8093-4b13593809db-istiod-ca-cert\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.836812    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-istio-token\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.836863    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-data\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.836899    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xqwj\" (UniqueName: \"kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-kube-api-access-7xqwj\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.836987    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-credential-socket\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.837142    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-socket\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.838038    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-certs\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.838110    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-data\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.838427    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-credential-socket\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.838775    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d7723e-9a7d-4858-8093-4b13593809db-istiod-ca-cert\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.840411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.839070    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-socket\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.866014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.862188    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d7723e-9a7d-4858-8093-4b13593809db-istio-podinfo\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.866268 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.866239    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xqwj\" (UniqueName: \"kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-kube-api-access-7xqwj\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.866859 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.866809    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-envoy\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:46.868487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:46.868460    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-istio-token\") pod \"e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") " pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:47.183096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:47.182320    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:47.754227 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:47.754187    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"]
Oct 30 00:20:47.754752 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:47.754353    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:47.754948 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:47.754911    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:47.755138 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:47.755106    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:20:48.031003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:48.030904    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" event={"ID":"71d7723e-9a7d-4858-8093-4b13593809db","Type":"ContainerStarted","Data":"e556283aabfe42c525e73a2e1fde7c897cbaa541a5c71f0adcdb2197190e53e9"}
Oct 30 00:20:48.031003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:48.030976    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" event={"ID":"71d7723e-9a7d-4858-8093-4b13593809db","Type":"ContainerStarted","Data":"65779125364d53a73f0b4882727b9ad525f421b496f94ef63f11849b6ba8a6b0"}
Oct 30 00:20:48.183367 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:48.183336    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:48.184649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:48.184623    4441 patch_prober.go:29] interesting pod/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 00:20:48.184795 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:48.184675    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" podUID="71d7723e-9a7d-4858-8093-4b13593809db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 00:20:49.188005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:49.187979    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:49.249021 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:49.248937    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" podStartSLOduration=3.248916492 podStartE2EDuration="3.248916492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:48.084380492 +0000 UTC m=+957.609674372" watchObservedRunningTime="2026-10-30 00:20:49.248916492 +0000 UTC m=+958.774210373"
Oct 30 00:20:50.046165 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:50.046129    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:50.047059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:50.047004    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:20:52.257546 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:52.257514    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"]
Oct 30 00:20:52.263737 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:52.263706    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"
Oct 30 00:20:52.282413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:52.282385    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"]
Oct 30 00:20:52.286279 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:52.286252    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e2a011\""
Oct 30 00:20:52.286569 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:52.286547    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e2a011-dockercfg-sm48w\""
Oct 30 00:20:52.432047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:52.430679    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8660a915-4fe9-4b55-bde2-30d2222da065-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h\" (UID: \"8660a915-4fe9-4b55-bde2-30d2222da065\") " pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"
Oct 30 00:20:52.432047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:52.430946    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcv4g\" (UniqueName: \"kubernetes.io/projected/8660a915-4fe9-4b55-bde2-30d2222da065-kube-api-access-tcv4g\") pod \"payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h\" (UID: \"8660a915-4fe9-4b55-bde2-30d2222da065\") " pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"
Oct 30 00:20:52.534484 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:52.534374    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcv4g\" (UniqueName: \"kubernetes.io/projected/8660a915-4fe9-4b55-bde2-30d2222da065-kube-api-access-tcv4g\") pod \"payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h\" (UID: \"8660a915-4fe9-4b55-bde2-30d2222da065\") " pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"
Oct 30 00:20:52.534941 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:52.534910    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8660a915-4fe9-4b55-bde2-30d2222da065-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h\" (UID: \"8660a915-4fe9-4b55-bde2-30d2222da065\") " pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"
Oct 30 00:20:52.543096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:52.536317    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8660a915-4fe9-4b55-bde2-30d2222da065-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h\" (UID: \"8660a915-4fe9-4b55-bde2-30d2222da065\") " pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"
Oct 30 00:20:52.587509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:52.587475    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcv4g\" (UniqueName: \"kubernetes.io/projected/8660a915-4fe9-4b55-bde2-30d2222da065-kube-api-access-tcv4g\") pod \"payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h\" (UID: \"8660a915-4fe9-4b55-bde2-30d2222da065\") " pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"
Oct 30 00:20:52.882665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:52.882583    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"
Oct 30 00:20:53.215632 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:53.215577    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"]
Oct 30 00:20:53.221866 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:20:53.221830    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8660a915_4fe9_4b55_bde2_30d2222da065.slice/crio-e28e37857cc7755f93312e68847b80953b9fa002dd52cdb9819cbf54e460e7d8 WatchSource:0}: Error finding container e28e37857cc7755f93312e68847b80953b9fa002dd52cdb9819cbf54e460e7d8: Status 404 returned error can't find the container with id e28e37857cc7755f93312e68847b80953b9fa002dd52cdb9819cbf54e460e7d8
Oct 30 00:20:54.118880 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:54.118838    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h" event={"ID":"8660a915-4fe9-4b55-bde2-30d2222da065","Type":"ContainerStarted","Data":"d62ef154bd0b9b95b4adb3e1b134936c5e10d7ee3f4b8507e57043525966788d"}
Oct 30 00:20:54.119537 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:54.119508    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h" event={"ID":"8660a915-4fe9-4b55-bde2-30d2222da065","Type":"ContainerStarted","Data":"e28e37857cc7755f93312e68847b80953b9fa002dd52cdb9819cbf54e460e7d8"}
Oct 30 00:20:54.120889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:54.120865    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"
Oct 30 00:20:54.198184 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:54.194888    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h" podStartSLOduration=2.194862095 podStartE2EDuration="2.194862095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:54.170379789 +0000 UTC m=+963.695673669" watchObservedRunningTime="2026-10-30 00:20:54.194862095 +0000 UTC m=+963.720155973"
Oct 30 00:20:55.309049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.296693    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"]
Oct 30 00:20:55.309049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.299114    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:55.338747 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.338705    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:55.420339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.420307    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"
Oct 30 00:20:55.473399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.455764    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-461609\""
Oct 30 00:20:55.473399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.455964    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-461609-dockercfg-dwzj9\""
Oct 30 00:20:55.493149 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.493107    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"]
Oct 30 00:20:55.547061 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.542626    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3de2b1e-69bd-4475-a45c-1904f032657c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8\" (UID: \"b3de2b1e-69bd-4475-a45c-1904f032657c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"
Oct 30 00:20:55.547061 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.542964    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjq4n\" (UniqueName: \"kubernetes.io/projected/b3de2b1e-69bd-4475-a45c-1904f032657c-kube-api-access-xjq4n\") pod \"payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8\" (UID: \"b3de2b1e-69bd-4475-a45c-1904f032657c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"
Oct 30 00:20:55.654104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.651832    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjq4n\" (UniqueName: \"kubernetes.io/projected/b3de2b1e-69bd-4475-a45c-1904f032657c-kube-api-access-xjq4n\") pod \"payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8\" (UID: \"b3de2b1e-69bd-4475-a45c-1904f032657c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"
Oct 30 00:20:55.654104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.652065    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3de2b1e-69bd-4475-a45c-1904f032657c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8\" (UID: \"b3de2b1e-69bd-4475-a45c-1904f032657c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"
Oct 30 00:20:55.654104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.653146    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3de2b1e-69bd-4475-a45c-1904f032657c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8\" (UID: \"b3de2b1e-69bd-4475-a45c-1904f032657c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"
Oct 30 00:20:55.752449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.745967    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjq4n\" (UniqueName: \"kubernetes.io/projected/b3de2b1e-69bd-4475-a45c-1904f032657c-kube-api-access-xjq4n\") pod \"payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8\" (UID: \"b3de2b1e-69bd-4475-a45c-1904f032657c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"
Oct 30 00:20:55.779295 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.775123    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"]
Oct 30 00:20:55.834703 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.804620    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"
Oct 30 00:20:55.865049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:55.864228    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:56.044504 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.040384    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"]
Oct 30 00:20:56.044504 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.040519    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"
Oct 30 00:20:56.072065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.050582    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"
Oct 30 00:20:56.117757 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.117716    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9wbr\" (UniqueName: \"kubernetes.io/projected/58f56459-1511-43bf-83bb-3fa7d5efeb9a-kube-api-access-x9wbr\") pod \"payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk\" (UID: \"58f56459-1511-43bf-83bb-3fa7d5efeb9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"
Oct 30 00:20:56.118120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.118085    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f56459-1511-43bf-83bb-3fa7d5efeb9a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk\" (UID: \"58f56459-1511-43bf-83bb-3fa7d5efeb9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"
Oct 30 00:20:56.371154 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.345994    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 00:20:56.372924 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.372896    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:20:56.373136 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.373094    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:56.400048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.399598    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f56459-1511-43bf-83bb-3fa7d5efeb9a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk\" (UID: \"58f56459-1511-43bf-83bb-3fa7d5efeb9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"
Oct 30 00:20:56.400048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.400006    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9wbr\" (UniqueName: \"kubernetes.io/projected/58f56459-1511-43bf-83bb-3fa7d5efeb9a-kube-api-access-x9wbr\") pod \"payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk\" (UID: \"58f56459-1511-43bf-83bb-3fa7d5efeb9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"
Oct 30 00:20:56.428050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.427875    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f56459-1511-43bf-83bb-3fa7d5efeb9a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk\" (UID: \"58f56459-1511-43bf-83bb-3fa7d5efeb9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"
Oct 30 00:20:56.435048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.428804    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:56.457059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.444202    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:56.457059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.429897    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:56.457679 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.457647    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:20:56.525893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.525856    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:20:56.526220 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.526169    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw" podUID="4489355f-aad2-4394-bd10-7a78024d6458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:56.555039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.554978    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:20:56.555317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.555271    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx" podUID="42d7f0fa-9352-4408-b1a1-cc80af286710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:56.574952 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.562298    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:56.828159 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.828085    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:20:56.829931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.829900    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9wbr\" (UniqueName: \"kubernetes.io/projected/58f56459-1511-43bf-83bb-3fa7d5efeb9a-kube-api-access-x9wbr\") pod \"payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk\" (UID: \"58f56459-1511-43bf-83bb-3fa7d5efeb9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"
Oct 30 00:20:56.947468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:56.947411    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:57.184037 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:57.164339    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:57.483222 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:57.456819    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:57.483222 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:57.428804    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:57.564185 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:57.564136    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:57.791834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:57.739934    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 00:20:57.908110 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:57.907953    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:20:58.070191 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:57.911866    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:58.322647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.279534    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:20:58.323530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.323115    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l" podUID="a70f3491-6506-456f-ab45-90b1291f086d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:58.386200 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.324088    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:58.505239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.478286    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:58.572179 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.572128    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:58.654104 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:20:58.651362    4441 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.621s"
Oct 30 00:20:58.750552 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.685474    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:20:58.801079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.753184    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:20:58.801079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.753267    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:20:58.801079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.753332    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:20:58.801079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.753383    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:20:58.801079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.753420    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:20:58.801079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.753479    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"
Oct 30 00:20:58.801079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.753549    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:20:58.848300 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.848257    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 00:20:58.898167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:58.898122    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:20:59.075053 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:59.039629    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:20:59.075053 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:59.039937    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 00:20:59.075053 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:59.073615    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 00:20:59.313213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:59.146905    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:20:59.313213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:59.146964    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z" podUID="30206a75-65df-4bc5-893f-5f4dc29a650c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:59.313213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:59.166742    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"
Oct 30 00:20:59.313213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:59.283339    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:59.485056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:59.480192    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:59.680063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:59.671302    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:20:59.701555 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:59.698537    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:20:59.789097 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:59.788760    4441 trace.go:236] Trace[1133135555]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-5vw7m" (30-Oct-2026 00:20:57.391) (total time: 2397ms):
Oct 30 00:20:59.789097 ip-10-0-137-69 kubenswrapper[4441]: Trace[1133135555]: [2.397243986s] [2.397243986s] END
Oct 30 00:20:59.857049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:59.850511    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:20:59.900731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:20:59.900690    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:21:00.047256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.047158    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:00.087261 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.051114    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:21:00.088652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.080302    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 00:21:00.295230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.291533    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:00.505673 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:21:00.504391    4441 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.571s"
Oct 30 00:21:00.505673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.504541    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8" event={"ID":"b3de2b1e-69bd-4475-a45c-1904f032657c","Type":"ContainerStarted","Data":"b91e27fa5812f6b37d280e53476d1aef5a2ad2b3d8935c3484212e8f7139786b"}
Oct 30 00:21:00.505673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.504678    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"]
Oct 30 00:21:00.505673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.504723    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"]
Oct 30 00:21:00.520867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.520836    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"]
Oct 30 00:21:00.521093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.520926    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:21:00.521093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.520966    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:21:00.521093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.521012    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:21:00.521093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.521059    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:00.521093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.521079    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:21:00.521655 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.521638    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:21:00.521936 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.521917    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:21:00.521991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.521977    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:21:00.548056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.543890    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7417dc-serving-cert\""
Oct 30 00:21:00.557127 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.522019    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:21:00.557127 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.552885    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:21:00.714142 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.714106    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3f287b-5853-45dc-a051-3238bdcdf0ed-openshift-service-ca\") pod \"maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:00.715741 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.714156    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-maas-api-tls\") pod \"maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:00.715741 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.714234    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p475n\" (UniqueName: \"kubernetes.io/projected/0c3f287b-5853-45dc-a051-3238bdcdf0ed-kube-api-access-p475n\") pod \"maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:00.715741 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.714333    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-metrics-certs\") pod \"maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:00.817875 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.817787    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3f287b-5853-45dc-a051-3238bdcdf0ed-openshift-service-ca\") pod \"maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:00.817875 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.817831    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-maas-api-tls\") pod \"maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:00.817875 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.817871    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p475n\" (UniqueName: \"kubernetes.io/projected/0c3f287b-5853-45dc-a051-3238bdcdf0ed-kube-api-access-p475n\") pod \"maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:00.818163 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.817953    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-metrics-certs\") pod \"maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:00.818672 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.818654    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3f287b-5853-45dc-a051-3238bdcdf0ed-openshift-service-ca\") pod \"maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:00.820850 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.820824    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-metrics-certs\") pod \"maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:00.864340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.864289    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-maas-api-tls\") pod \"maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:00.892647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.883712    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"]
Oct 30 00:21:00.892647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:00.887836    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p475n\" (UniqueName: \"kubernetes.io/projected/0c3f287b-5853-45dc-a051-3238bdcdf0ed-kube-api-access-p475n\") pod \"maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:00.899733 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:21:00.899709    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f56459_1511_43bf_83bb_3fa7d5efeb9a.slice/crio-8a47b9002be265b7c721ea0cda35cac6833dc99f63b8f048bf10bb7c8eafd139 WatchSource:0}: Error finding container 8a47b9002be265b7c721ea0cda35cac6833dc99f63b8f048bf10bb7c8eafd139: Status 404 returned error can't find the container with id 8a47b9002be265b7c721ea0cda35cac6833dc99f63b8f048bf10bb7c8eafd139
Oct 30 00:21:01.178590 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:01.178463    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:01.512097 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:01.512058    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8" event={"ID":"b3de2b1e-69bd-4475-a45c-1904f032657c","Type":"ContainerStarted","Data":"2e130ddd96a7537df959184a2c3ff2512b146ccf255e197d8c8e583f70bedef6"}
Oct 30 00:21:01.512296 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:01.512141    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"
Oct 30 00:21:01.513745 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:01.513720    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk" event={"ID":"58f56459-1511-43bf-83bb-3fa7d5efeb9a","Type":"ContainerStarted","Data":"d3afb2dba2452d2e4b75606fdbad53a3efbc41665fa3f129a3eabbf71442f52c"}
Oct 30 00:21:01.513852 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:01.513769    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk" event={"ID":"58f56459-1511-43bf-83bb-3fa7d5efeb9a","Type":"ContainerStarted","Data":"8a47b9002be265b7c721ea0cda35cac6833dc99f63b8f048bf10bb7c8eafd139"}
Oct 30 00:21:01.513906 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:01.513884    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"
Oct 30 00:21:01.556542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:01.556515    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"]
Oct 30 00:21:01.557012 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:21:01.556988    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3f287b_5853_45dc_a051_3238bdcdf0ed.slice/crio-156249bc988ecf00f1d9df821a1a7f84902c165282ab5cd8f5f9441a01569d20 WatchSource:0}: Error finding container 156249bc988ecf00f1d9df821a1a7f84902c165282ab5cd8f5f9441a01569d20: Status 404 returned error can't find the container with id 156249bc988ecf00f1d9df821a1a7f84902c165282ab5cd8f5f9441a01569d20
Oct 30 00:21:01.650051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:01.642961    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8" podStartSLOduration=6.642941978 podStartE2EDuration="6.642941978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:01.641314915 +0000 UTC m=+971.166608826" watchObservedRunningTime="2026-10-30 00:21:01.642941978 +0000 UTC m=+971.168235858"
Oct 30 00:21:01.684751 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:01.684692    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk" podStartSLOduration=6.684674616 podStartE2EDuration="6.684674616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:01.678667573 +0000 UTC m=+971.203961455" watchObservedRunningTime="2026-10-30 00:21:01.684674616 +0000 UTC m=+971.209968494"
Oct 30 00:21:02.563855 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:02.563812    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l" event={"ID":"0c3f287b-5853-45dc-a051-3238bdcdf0ed","Type":"ContainerStarted","Data":"26c4219d664546e7f975c58f0b80489ae0f0698d5554cbcaf7effddd1e2fc85d"}
Oct 30 00:21:02.564342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:02.563884    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l" event={"ID":"0c3f287b-5853-45dc-a051-3238bdcdf0ed","Type":"ContainerStarted","Data":"156249bc988ecf00f1d9df821a1a7f84902c165282ab5cd8f5f9441a01569d20"}
Oct 30 00:21:02.564342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:02.563917    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:02.608480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:02.608429    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l" podStartSLOduration=4.160999719 podStartE2EDuration="4.608415265s" totalImagesPullingTime="447.415546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:02.6041053 +0000 UTC m=+972.129399181" watchObservedRunningTime="2026-10-30 00:21:02.608415265 +0000 UTC m=+972.133709141"
Oct 30 00:21:03.901373 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:21:03.901338    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:21:05.359867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.359828    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"]
Oct 30 00:21:05.365116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.365086    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.412261 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.411516    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"]
Oct 30 00:21:05.433085 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.428789    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.433085 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.428843    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.433085 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.428875    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.433085 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.428969    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dq4t\" (UniqueName: \"kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-kube-api-access-4dq4t\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.433085 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.429008    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.433085 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.429214    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.433085 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.429267    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.433085 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.429499    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.433085 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.429562    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.531266 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.531224    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dq4t\" (UniqueName: \"kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-kube-api-access-4dq4t\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.531451 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.531290    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.531451 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.531353    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.531451 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.531396    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.531748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.531605    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.531748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.531665    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.532285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.532058    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.532285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.532098    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.532285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.532126    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.532495 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.532279    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.532550 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.532534    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.533168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.533148    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.533308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.533234    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.533484 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.533294    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.535575 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.535548    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.535856 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.535832    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.561771 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.561739    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dq4t\" (UniqueName: \"kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-kube-api-access-4dq4t\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.562806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.562782    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:05.686572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:05.686487    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:06.070075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:06.062256    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"]
Oct 30 00:21:06.081165 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:06.081125    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:21:06.081289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:06.081215    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:21:06.081289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:06.081265    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:21:06.613418 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:06.613384    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8" event={"ID":"a3ba27e0-c510-41a6-97ec-83d0c5b637ec","Type":"ContainerStarted","Data":"22b3d74c6ddee56cd5a91906871a8e13ea445db114cb04c2a21c454984bf84da"}
Oct 30 00:21:06.613798 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:06.613447    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8" event={"ID":"a3ba27e0-c510-41a6-97ec-83d0c5b637ec","Type":"ContainerStarted","Data":"f312c143b2d73c550bb31a90b6cb5dc7fa3cbf13703ffd1c35d098aa8e3c3e34"}
Oct 30 00:21:06.665654 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:06.665591    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8" podStartSLOduration=1.665574809 podStartE2EDuration="1.665574809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:06.662228505 +0000 UTC m=+976.187522384" watchObservedRunningTime="2026-10-30 00:21:06.665574809 +0000 UTC m=+976.190868686"
Oct 30 00:21:06.687495 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:06.687465    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:07.688718 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:07.688688    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:21:07.689304 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:07.688748    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8" podUID="a3ba27e0-c510-41a6-97ec-83d0c5b637ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:21:07.947048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:07.946957    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"]
Oct 30 00:21:07.981974 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:07.981935    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"]
Oct 30 00:21:07.982182 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:07.981983    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.001332 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.001287    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e2a011-serving-cert\""
Oct 30 00:21:08.081521 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.081422    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2fd350b-e02c-488b-af30-a192c3d9c53f-openshift-service-ca\") pod \"maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.082010 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.081977    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-maas-api-tls\") pod \"maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.082175 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.082158    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkczg\" (UniqueName: \"kubernetes.io/projected/e2fd350b-e02c-488b-af30-a192c3d9c53f-kube-api-access-vkczg\") pod \"maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.082367 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.082350    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-metrics-certs\") pod \"maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.184708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.184671    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-maas-api-tls\") pod \"maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.184973 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.184945    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkczg\" (UniqueName: \"kubernetes.io/projected/e2fd350b-e02c-488b-af30-a192c3d9c53f-kube-api-access-vkczg\") pod \"maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.185254 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.185232    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-metrics-certs\") pod \"maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.185565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.185541    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2fd350b-e02c-488b-af30-a192c3d9c53f-openshift-service-ca\") pod \"maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.187225 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.186950    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2fd350b-e02c-488b-af30-a192c3d9c53f-openshift-service-ca\") pod \"maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.193296 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.193245    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-maas-api-tls\") pod \"maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.205072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.204946    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-metrics-certs\") pod \"maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.210955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.210912    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkczg\" (UniqueName: \"kubernetes.io/projected/e2fd350b-e02c-488b-af30-a192c3d9c53f-kube-api-access-vkczg\") pod \"maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.360413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.360373    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:08.582239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.581628    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:21:08.703618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.688601    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:21:08.703618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.688687    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8" podUID="a3ba27e0-c510-41a6-97ec-83d0c5b637ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:21:08.780510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:08.779557    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"]
Oct 30 00:21:09.219471 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.212638    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"]
Oct 30 00:21:09.232291 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.223777    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.285052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.280700    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"]
Oct 30 00:21:09.312585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.312534    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.312833 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.312814    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.312955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.312941    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.313099 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.313084    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx2nq\" (UniqueName: \"kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-kube-api-access-rx2nq\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.317415 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.317393    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.317686 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.317665    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.317911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.317893    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.318009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.317996    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.318299 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.318280    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.419967 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.419930    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.420202 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.419989    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.420202 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.420014    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.420390 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.420365    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.420390 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.420376    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.420550 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.420465    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.420616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.420585    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.420704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.420685    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.420795 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.420764    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.420854 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.420798    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.420911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.420860    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2nq\" (UniqueName: \"kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-kube-api-access-rx2nq\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.420911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.420889    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.432054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.429083    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.432054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.431698    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.440046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.437668    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.468047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.467981    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.481145 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.476420    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.487809 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.487746    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2nq\" (UniqueName: \"kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-kube-api-access-rx2nq\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.626341 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.626102    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:09.659256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.659220    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr" event={"ID":"e2fd350b-e02c-488b-af30-a192c3d9c53f","Type":"ContainerStarted","Data":"97fdc76a71dc2301b57b2014c1b9bc64c86ef6fd132b8217cdeaecdad0e88d27"}
Oct 30 00:21:09.660601 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.660576    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr" event={"ID":"e2fd350b-e02c-488b-af30-a192c3d9c53f","Type":"ContainerStarted","Data":"51a2e05f829bd3932b2b999a6a5a838aec47a2b419d4e44811f27a461bdbbf27"}
Oct 30 00:21:09.660800 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.660786    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:09.688166 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.687833    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:21:09.688166 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.687875    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8" podUID="a3ba27e0-c510-41a6-97ec-83d0c5b637ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:21:09.708503 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:09.708448    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr" podStartSLOduration=2.169572843 podStartE2EDuration="2.708429742s" totalImagesPullingTime="538.856899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:09.702441366 +0000 UTC m=+979.227735244" watchObservedRunningTime="2026-10-30 00:21:09.708429742 +0000 UTC m=+979.233723622"
Oct 30 00:21:10.073100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.070562    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"]
Oct 30 00:21:10.105352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.104810    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:21:10.105352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.104913    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:21:10.105352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.104954    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:21:10.692321 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.691949    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:21:10.692321 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.692014    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8" podUID="a3ba27e0-c510-41a6-97ec-83d0c5b637ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:21:10.710461 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.710431    4441 generic.go:356] "Generic (PLEG): container finished" podID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerID="31080c30ea6d0f0fbca5e10172fc2bbdb3f4bddd766d5a78c86fcc66e5b188b9" exitCode=0
Oct 30 00:21:10.710839 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.710508    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" event={"ID":"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5","Type":"ContainerDied","Data":"31080c30ea6d0f0fbca5e10172fc2bbdb3f4bddd766d5a78c86fcc66e5b188b9"}
Oct 30 00:21:10.711089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.711067    4441 scope.go:122] "RemoveContainer" containerID="31080c30ea6d0f0fbca5e10172fc2bbdb3f4bddd766d5a78c86fcc66e5b188b9"
Oct 30 00:21:10.713244 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.713207    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:21:10.713331 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.713299    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:21:10.713374 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.713345    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:21:10.713467 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.713441    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd" event={"ID":"2ae884f6-aa1a-4aca-addf-a0c37c26aa80","Type":"ContainerStarted","Data":"7642a7dcb345247fd59c1df969cdb375e69e5edda154845f359a47783367470a"}
Oct 30 00:21:10.713534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.713498    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd" event={"ID":"2ae884f6-aa1a-4aca-addf-a0c37c26aa80","Type":"ContainerStarted","Data":"bf8354449116dc8aec09f35624539f5975cd59d4af3d7729695f67999c9b2e4b"}
Oct 30 00:21:10.814790 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:10.814730    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd" podStartSLOduration=1.8147082939999999 podStartE2EDuration="1.814708294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:10.81248669 +0000 UTC m=+980.337780568" watchObservedRunningTime="2026-10-30 00:21:10.814708294 +0000 UTC m=+980.340002172"
Oct 30 00:21:11.630048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:11.629188    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:11.644109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:11.638161    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 00:21:11.644109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:11.638238    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd" podUID="2ae884f6-aa1a-4aca-addf-a0c37c26aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 00:21:11.693715 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:11.693665    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:21:11.693909 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:11.693734    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8" podUID="a3ba27e0-c510-41a6-97ec-83d0c5b637ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:21:11.770052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:11.765855    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" event={"ID":"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5","Type":"ContainerStarted","Data":"e742d6d453a430097636f549e1de20335941b668c586f7d071b1969e85981e89"}
Oct 30 00:21:12.015068 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.011408    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"]
Oct 30 00:21:12.101047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.092419    4441 patch_prober.go:29] interesting pod/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:21:12.101047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.092505    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:21:12.421159 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.394521    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:21:12.421159 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.394651    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:12.421159 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.411992    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-461609-serving-cert\""
Oct 30 00:21:12.427278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.427233    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"]
Oct 30 00:21:12.572777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.552475    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-maas-api-tls\") pod \"maas-api-e2e-api-a-461609-6b675c587b-58jkr\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:12.572777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.552630    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-metrics-certs\") pod \"maas-api-e2e-api-a-461609-6b675c587b-58jkr\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:12.572777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.552799    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29vmk\" (UniqueName: \"kubernetes.io/projected/32724ded-d541-4e35-bd55-58b781767acb-kube-api-access-29vmk\") pod \"maas-api-e2e-api-a-461609-6b675c587b-58jkr\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:12.572777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.552948    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32724ded-d541-4e35-bd55-58b781767acb-openshift-service-ca\") pod \"maas-api-e2e-api-a-461609-6b675c587b-58jkr\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:12.613072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.608258    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"
Oct 30 00:21:12.658585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.658546    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-maas-api-tls\") pod \"maas-api-e2e-api-a-461609-6b675c587b-58jkr\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:12.658950 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.658929    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-metrics-certs\") pod \"maas-api-e2e-api-a-461609-6b675c587b-58jkr\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:12.659263 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.659245    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29vmk\" (UniqueName: \"kubernetes.io/projected/32724ded-d541-4e35-bd55-58b781767acb-kube-api-access-29vmk\") pod \"maas-api-e2e-api-a-461609-6b675c587b-58jkr\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:12.661566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.661517    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 00:21:12.661774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.661750    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd" podUID="2ae884f6-aa1a-4aca-addf-a0c37c26aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 00:21:12.683608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.683527    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32724ded-d541-4e35-bd55-58b781767acb-openshift-service-ca\") pod \"maas-api-e2e-api-a-461609-6b675c587b-58jkr\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:12.685361 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.685335    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32724ded-d541-4e35-bd55-58b781767acb-openshift-service-ca\") pod \"maas-api-e2e-api-a-461609-6b675c587b-58jkr\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:12.702500 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.702468    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:21:12.703173 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.703144    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8" podUID="a3ba27e0-c510-41a6-97ec-83d0c5b637ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:21:12.754149 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.754120    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"
Oct 30 00:21:12.755530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.755172    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vmk\" (UniqueName: \"kubernetes.io/projected/32724ded-d541-4e35-bd55-58b781767acb-kube-api-access-29vmk\") pod \"maas-api-e2e-api-a-461609-6b675c587b-58jkr\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:12.816159 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.766795    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-maas-api-tls\") pod \"maas-api-e2e-api-a-461609-6b675c587b-58jkr\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:12.816159 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.793542    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-metrics-certs\") pod \"maas-api-e2e-api-a-461609-6b675c587b-58jkr\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:12.910042 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.909597    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:21:12.939095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:12.925153    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:13.114605 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:13.110254    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:13.114933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:13.114906    4441 patch_prober.go:29] interesting pod/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:21:13.115125 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:13.115097    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:21:13.271111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:13.216898    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 00:21:13.795761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:13.795723    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 00:21:13.796100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:13.796062    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd" podUID="2ae884f6-aa1a-4aca-addf-a0c37c26aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 00:21:13.801559 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:13.797233    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:21:13.801975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:13.801727    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8" podUID="a3ba27e0-c510-41a6-97ec-83d0c5b637ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:21:14.088421 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:14.088382    4441 patch_prober.go:29] interesting pod/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:21:14.089038 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:14.088989    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:21:14.120007 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:14.119965    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"]
Oct 30 00:21:14.420201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:14.420159    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:14.633977 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:14.633892    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 00:21:14.634254 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:14.634222    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd" podUID="2ae884f6-aa1a-4aca-addf-a0c37c26aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 00:21:15.283059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:15.272277    4441 patch_prober.go:29] interesting pod/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:21:15.283059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:15.272397    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:21:15.483882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:15.483363    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:15.485191 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:15.484435    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:15.496479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:15.496454    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:21:15.497349 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:15.497319    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr" event={"ID":"32724ded-d541-4e35-bd55-58b781767acb","Type":"ContainerStarted","Data":"4597d3c82918e0329602b151dedaf1904f0f8b7e753198477d1c11df96d9a734"}
Oct 30 00:21:15.652100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:15.649976    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 00:21:15.652397 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:15.652360    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd" podUID="2ae884f6-aa1a-4aca-addf-a0c37c26aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 00:21:16.100081 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.099772    4441 patch_prober.go:29] interesting pod/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:21:16.100081 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.099837    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:21:16.507216 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.506971    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr" event={"ID":"32724ded-d541-4e35-bd55-58b781767acb","Type":"ContainerStarted","Data":"c66a0c60a153249d3d93c831c3fab338a583c8189c63e757ac82eb80a1c3694c"}
Oct 30 00:21:16.507216 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.507164    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:16.600725 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.600693    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"]
Oct 30 00:21:16.605950 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.605891    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr" podStartSLOduration=4.726111734 podStartE2EDuration="5.60587359s" totalImagesPullingTime="879.761856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:16.576472466 +0000 UTC m=+986.101766346" watchObservedRunningTime="2026-10-30 00:21:16.60587359 +0000 UTC m=+986.131167468"
Oct 30 00:21:16.610318 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.610293    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.630660 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.630633    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 00:21:16.630869 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.630840    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd" podUID="2ae884f6-aa1a-4aca-addf-a0c37c26aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 00:21:16.632255 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.632232    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nht5w\""
Oct 30 00:21:16.674043 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.649342    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"]
Oct 30 00:21:16.674311 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.672313    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 00:21:16.674422 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.672368    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 00:21:16.674509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.672406    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 00:21:16.688831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.686817    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04666c76-7d71-480d-bae1-27c724358c54-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.688831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.686882    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04666c76-7d71-480d-bae1-27c724358c54-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.688831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.686979    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04666c76-7d71-480d-bae1-27c724358c54-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.688831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.687013    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04666c76-7d71-480d-bae1-27c724358c54-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.688831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.687175    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04666c76-7d71-480d-bae1-27c724358c54-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.688831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.687239    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04666c76-7d71-480d-bae1-27c724358c54-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.688831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.687342    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6lds\" (UniqueName: \"kubernetes.io/projected/04666c76-7d71-480d-bae1-27c724358c54-kube-api-access-m6lds\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.895474 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.895367    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04666c76-7d71-480d-bae1-27c724358c54-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.895474 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.895454    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04666c76-7d71-480d-bae1-27c724358c54-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.895724 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.895591    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04666c76-7d71-480d-bae1-27c724358c54-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.895724 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.895629    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04666c76-7d71-480d-bae1-27c724358c54-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.901255 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.901224    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:21:16.919975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.919939    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04666c76-7d71-480d-bae1-27c724358c54-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.920338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.920315    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04666c76-7d71-480d-bae1-27c724358c54-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.925070 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.921112    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6lds\" (UniqueName: \"kubernetes.io/projected/04666c76-7d71-480d-bae1-27c724358c54-kube-api-access-m6lds\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.929186 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.922087    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04666c76-7d71-480d-bae1-27c724358c54-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.937080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.937057    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04666c76-7d71-480d-bae1-27c724358c54-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:16.997072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:16.978516    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04666c76-7d71-480d-bae1-27c724358c54-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:17.008780 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:17.008361    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04666c76-7d71-480d-bae1-27c724358c54-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:17.018957 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:17.018559    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04666c76-7d71-480d-bae1-27c724358c54-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:17.020567 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:17.020424    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04666c76-7d71-480d-bae1-27c724358c54-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:17.093049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:17.087476    4441 patch_prober.go:29] interesting pod/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:21:17.093049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:17.087554    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:21:17.111073 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:17.110838    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6lds\" (UniqueName: \"kubernetes.io/projected/04666c76-7d71-480d-bae1-27c724358c54-kube-api-access-m6lds\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2c22\" (UID: \"04666c76-7d71-480d-bae1-27c724358c54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:17.301721 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:17.301682    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:17.666100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:17.626944    4441 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 00:21:17.667046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:17.666987    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd" podUID="2ae884f6-aa1a-4aca-addf-a0c37c26aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 00:21:17.714921 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:17.714891    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"]
Oct 30 00:21:17.715586 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:21:17.715551    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04666c76_7d71_480d_bae1_27c724358c54.slice/crio-2ad41056405350e9e156fd166360f5739b1a3bfd947b62d27c0355205f6d5dad WatchSource:0}: Error finding container 2ad41056405350e9e156fd166360f5739b1a3bfd947b62d27c0355205f6d5dad: Status 404 returned error can't find the container with id 2ad41056405350e9e156fd166360f5739b1a3bfd947b62d27c0355205f6d5dad
Oct 30 00:21:18.091594 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:18.091563    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:21:18.092980 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:18.092954    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:21:18.731050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:18.730379    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22" event={"ID":"04666c76-7d71-480d-bae1-27c724358c54","Type":"ContainerStarted","Data":"2ad41056405350e9e156fd166360f5739b1a3bfd947b62d27c0355205f6d5dad"}
Oct 30 00:21:18.937047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:18.936534    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:21:19.160098 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:19.159733    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:19.171049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:19.161476    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:19.199051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:19.198509    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:21:22.364050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:22.350334    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:22.713248 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:22.713153    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:22.713772 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:22.713741    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:22.909157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:22.909106    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:22.932109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:22.925813    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:23.065394 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.065347    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:21:23.284585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.284535    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:23.285971 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.285625    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:23.404277 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.404182    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:23.465080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.445447    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:23.555057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.549866    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:23.577671 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.577637    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:21:23.651473 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.651440    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:21:23.732049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.728721    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:23.754780 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.737484    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:21:23.754780 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.740902    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:21:23.754780 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.740960    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:21:23.913051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.909734    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:23.918932 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:23.917655    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:21:24.047478 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.047391    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 00:21:24.061305 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.061117    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:21:24.085800 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.082120    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:24.085800 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.082458    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:21:24.268317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.268278    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:21:24.268607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.268572    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:24.287161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.287102    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 00:21:24.294056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.292933    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:21:24.337632 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.337536    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:24.418056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.410449    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:24.437201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.434272    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:24.647108 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.613734    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:24.760419 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.760371    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:24.920543 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:24.919135    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:25.075436 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:25.071561    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 00:21:25.086227 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:25.086181    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:25.190401 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:25.190316    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:21:25.190661 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:25.190648    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:21:25.190805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:25.190795    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:21:25.190928 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:25.190919    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:21:25.191057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:25.191045    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:21:25.191221 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:25.191211    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:21:25.191342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:25.191333    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:21:25.200757 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:25.200730    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:21:25.202177 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:25.202150    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:21:26.546038 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:26.545989    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:21:26.546338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:26.546129    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:21:26.867348 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:26.867266    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22" event={"ID":"04666c76-7d71-480d-bae1-27c724358c54","Type":"ContainerStarted","Data":"aba5028b0d57df93ca00a0256d9f4dc919b474931e605fe4cf440983a84f3a37"}
Oct 30 00:21:26.867531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:26.867374    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:26.933366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:26.933299    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22" podStartSLOduration=2.10707458 podStartE2EDuration="10.933278391s" totalImagesPullingTime="8.826203811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:26.920706712 +0000 UTC m=+996.446000589" watchObservedRunningTime="2026-10-30 00:21:26.933278391 +0000 UTC m=+996.458572269"
Oct 30 00:21:27.887980 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:27.887951    4441 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-l2c22 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:21:27.888504 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:27.888475    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22" podUID="04666c76-7d71-480d-bae1-27c724358c54" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:21:30.887336 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:30.887306    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2c22"
Oct 30 00:21:33.923452 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:21:33.923406    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:21:40.464572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:40.464537    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"]
Oct 30 00:21:40.507157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:40.507124    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"]
Oct 30 00:21:45.465291 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:45.465236    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2" podUID="94a58ad1-9196-49a3-a61a-a3e7ebb5cc01" containerName="payload-processing" containerID="cri-o://f1cc0158c5b1bc07947ac340c2f71ae644719062e37873ed86dfccd204ed0993" gracePeriod=25
Oct 30 00:21:45.507659 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:45.507615    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f" podUID="13255825-7146-4789-9262-3cb401f87bd3" containerName="payload-pre-processing" containerID="cri-o://6ab4630f8fc8445453b910ba6d03a9cdd76ede98e7c852a8ed901822d057b00a" gracePeriod=25
Oct 30 00:21:45.847303 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:45.847270    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"
Oct 30 00:21:45.939220 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:45.939007    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-kube-api-access-djxg8\" (UniqueName: \"kubernetes.io/projected/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-kube-api-access-djxg8\") pod \"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01\" (UID: \"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01\") "
Oct 30 00:21:45.939444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:45.939296    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-plugins-config-volume\") pod \"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01\" (UID: \"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01\") "
Oct 30 00:21:45.951236 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:45.940665    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-plugins-config-volume" pod "94a58ad1-9196-49a3-a61a-a3e7ebb5cc01" (UID: "94a58ad1-9196-49a3-a61a-a3e7ebb5cc01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:21:45.958764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:45.958725    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-kube-api-access-djxg8" pod "94a58ad1-9196-49a3-a61a-a3e7ebb5cc01" (UID: "94a58ad1-9196-49a3-a61a-a3e7ebb5cc01"). InnerVolumeSpecName "kube-api-access-djxg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:21:45.992600 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:45.992566    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"
Oct 30 00:21:46.045095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.041792    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13255825-7146-4789-9262-3cb401f87bd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13255825-7146-4789-9262-3cb401f87bd3-plugins-config-volume\") pod \"13255825-7146-4789-9262-3cb401f87bd3\" (UID: \"13255825-7146-4789-9262-3cb401f87bd3\") "
Oct 30 00:21:46.045095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.041986    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13255825-7146-4789-9262-3cb401f87bd3-kube-api-access-xmvjv\" (UniqueName: \"kubernetes.io/projected/13255825-7146-4789-9262-3cb401f87bd3-kube-api-access-xmvjv\") pod \"13255825-7146-4789-9262-3cb401f87bd3\" (UID: \"13255825-7146-4789-9262-3cb401f87bd3\") "
Oct 30 00:21:46.045095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.042715    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:21:46.045095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.042740    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djxg8\" (UniqueName: \"kubernetes.io/projected/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01-kube-api-access-djxg8\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:21:46.045095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.043960    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13255825-7146-4789-9262-3cb401f87bd3-plugins-config-volume" pod "13255825-7146-4789-9262-3cb401f87bd3" (UID: "13255825-7146-4789-9262-3cb401f87bd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:21:46.049663 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.049624    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13255825-7146-4789-9262-3cb401f87bd3-kube-api-access-xmvjv" pod "13255825-7146-4789-9262-3cb401f87bd3" (UID: "13255825-7146-4789-9262-3cb401f87bd3"). InnerVolumeSpecName "kube-api-access-xmvjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:21:46.089563 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.089521    4441 generic.go:356] "Generic (PLEG): container finished" podID="13255825-7146-4789-9262-3cb401f87bd3" containerID="6ab4630f8fc8445453b910ba6d03a9cdd76ede98e7c852a8ed901822d057b00a" exitCode=0
Oct 30 00:21:46.089563 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.089570    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"
Oct 30 00:21:46.089817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.089683    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f" event={"ID":"13255825-7146-4789-9262-3cb401f87bd3","Type":"ContainerDied","Data":"6ab4630f8fc8445453b910ba6d03a9cdd76ede98e7c852a8ed901822d057b00a"}
Oct 30 00:21:46.089817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.089735    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f" event={"ID":"13255825-7146-4789-9262-3cb401f87bd3","Type":"ContainerDied","Data":"048a30e97261bf9f9ee66eaf390d8a0ffcd88854d4f59b254c8c0d358d06d35f"}
Oct 30 00:21:46.089817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.089765    4441 scope.go:122] "RemoveContainer" containerID="6ab4630f8fc8445453b910ba6d03a9cdd76ede98e7c852a8ed901822d057b00a"
Oct 30 00:21:46.094686 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.094658    4441 generic.go:356] "Generic (PLEG): container finished" podID="94a58ad1-9196-49a3-a61a-a3e7ebb5cc01" containerID="f1cc0158c5b1bc07947ac340c2f71ae644719062e37873ed86dfccd204ed0993" exitCode=0
Oct 30 00:21:46.094784 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.094731    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2" event={"ID":"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01","Type":"ContainerDied","Data":"f1cc0158c5b1bc07947ac340c2f71ae644719062e37873ed86dfccd204ed0993"}
Oct 30 00:21:46.094784 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.094753    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"
Oct 30 00:21:46.094784 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.094777    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2" event={"ID":"94a58ad1-9196-49a3-a61a-a3e7ebb5cc01","Type":"ContainerDied","Data":"9a8008f8e919a105ead1b0aa3c4ece392876701b5fb5b0509e25b2edaea3b0e6"}
Oct 30 00:21:46.113723 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.113702    4441 scope.go:122] "RemoveContainer" containerID="6ab4630f8fc8445453b910ba6d03a9cdd76ede98e7c852a8ed901822d057b00a"
Oct 30 00:21:46.115074 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:21:46.115005    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab4630f8fc8445453b910ba6d03a9cdd76ede98e7c852a8ed901822d057b00a\": container with ID starting with 6ab4630f8fc8445453b910ba6d03a9cdd76ede98e7c852a8ed901822d057b00a not found: ID does not exist" containerID="6ab4630f8fc8445453b910ba6d03a9cdd76ede98e7c852a8ed901822d057b00a"
Oct 30 00:21:46.115190 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.115086    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab4630f8fc8445453b910ba6d03a9cdd76ede98e7c852a8ed901822d057b00a"} err="failed to get container status \"6ab4630f8fc8445453b910ba6d03a9cdd76ede98e7c852a8ed901822d057b00a\": rpc error: code = NotFound desc = could not find container \"6ab4630f8fc8445453b910ba6d03a9cdd76ede98e7c852a8ed901822d057b00a\": container with ID starting with 6ab4630f8fc8445453b910ba6d03a9cdd76ede98e7c852a8ed901822d057b00a not found: ID does not exist"
Oct 30 00:21:46.115190 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.115111    4441 scope.go:122] "RemoveContainer" containerID="f1cc0158c5b1bc07947ac340c2f71ae644719062e37873ed86dfccd204ed0993"
Oct 30 00:21:46.138362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.137069    4441 scope.go:122] "RemoveContainer" containerID="f1cc0158c5b1bc07947ac340c2f71ae644719062e37873ed86dfccd204ed0993"
Oct 30 00:21:46.140688 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:21:46.140558    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1cc0158c5b1bc07947ac340c2f71ae644719062e37873ed86dfccd204ed0993\": container with ID starting with f1cc0158c5b1bc07947ac340c2f71ae644719062e37873ed86dfccd204ed0993 not found: ID does not exist" containerID="f1cc0158c5b1bc07947ac340c2f71ae644719062e37873ed86dfccd204ed0993"
Oct 30 00:21:46.140688 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.140617    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cc0158c5b1bc07947ac340c2f71ae644719062e37873ed86dfccd204ed0993"} err="failed to get container status \"f1cc0158c5b1bc07947ac340c2f71ae644719062e37873ed86dfccd204ed0993\": rpc error: code = NotFound desc = could not find container \"f1cc0158c5b1bc07947ac340c2f71ae644719062e37873ed86dfccd204ed0993\": container with ID starting with f1cc0158c5b1bc07947ac340c2f71ae644719062e37873ed86dfccd204ed0993 not found: ID does not exist"
Oct 30 00:21:46.146630 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.146601    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13255825-7146-4789-9262-3cb401f87bd3-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:21:46.146797 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.146781    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmvjv\" (UniqueName: \"kubernetes.io/projected/13255825-7146-4789-9262-3cb401f87bd3-kube-api-access-xmvjv\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:21:46.164149 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.164119    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"]
Oct 30 00:21:46.171209 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.171177    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3fae6d-7d89bf4647-kdx5f"]
Oct 30 00:21:46.197563 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.197530    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"]
Oct 30 00:21:46.201625 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.201595    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3fae6d-6467bd7f48-hwhp2"]
Oct 30 00:21:46.634189 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.634151    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13255825-7146-4789-9262-3cb401f87bd3" path="/var/lib/kubelet/pods/13255825-7146-4789-9262-3cb401f87bd3/volumes"
Oct 30 00:21:46.634861 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:46.634841    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a58ad1-9196-49a3-a61a-a3e7ebb5cc01" path="/var/lib/kubelet/pods/94a58ad1-9196-49a3-a61a-a3e7ebb5cc01/volumes"
Oct 30 00:21:59.741113 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:59.741070    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:21:59.766531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:21:59.766486    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:00.096205 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:00.096106    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:00.128497 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:00.096796    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:00.129829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:00.129515    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:00.184667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:00.183771    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:00.214791 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:00.214649    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:22:00.422861 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:00.422746    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:00.752048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:00.747602    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 00:22:00.788049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:00.782100    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:00.796047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:00.789680    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:22:01.105108 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.104651    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:01.137808 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.137757    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:01.139348 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.139313    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:01.158625 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.149300    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:01.158625 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.150772    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 00:22:01.365101 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.364395    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:22:01.375047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.370353    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:22:01.418545 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.418513    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:22:01.418833 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.418814    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:22:01.418987 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.418974    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:22:01.419146 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.419132    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:22:01.419284 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.419272    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:22:01.419403 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.419392    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:22:01.419520 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.419508    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:22:01.501052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.442967    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:01.643281 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.503740    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:22:01.759062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.751879    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:01.794947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.794901    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:01.795910 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.795882    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:22:01.865082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:01.856210    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:02.167048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:02.147088    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:02.167048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:02.153471    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:02.169320 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:02.169280    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 00:22:02.310690 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:02.310654    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:22:02.340875 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:02.340848    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:22:02.388790 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:02.388727    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:02.390674 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:02.390636    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 30 00:22:02.390974 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:02.390947    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:02.791046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:02.765454    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 00:22:03.060847 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:02.795418    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:03.108050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:02.923419    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:03.353561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:03.353463    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:03.531683 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:03.531643    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:03.541055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:03.535481    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:03.929090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:03.928865    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:04.171373 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:22:04.171332    4441 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.557s"
Oct 30 00:22:04.175253 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.175228    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:22:04.175528 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.175495    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:22:04.217440 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.217339    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:04.264060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.179176    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8" podUID="b3de2b1e-69bd-4475-a45c-1904f032657c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.95:50052\" within 1s: context deadline exceeded"
Oct 30 00:22:04.264060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.228872    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hx728" podUID="7100f212-f2df-48aa-9895-d51333bc10ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:04.302610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.302550    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:22:04.303138 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.303109    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:22:04.303447 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.303425    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:22:04.366154 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.366108    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:22:04.377059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.370105    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:22:04.411156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.409073    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:04.588171 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.588127    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:04.596159 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.596114    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:04.596680 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:04.596654    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:22:05.320425 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:05.318114    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:05.368102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:05.331632    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:22:05.368400 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:05.368356    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:22:05.613364 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:22:05.613276    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:22:05.636062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:05.632849    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:22:05.770047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:05.690645    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:22:05.770047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:05.730451    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded"
Oct 30 00:22:05.789049 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:22:05.786471    4441 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.16s"
Oct 30 00:22:05.789049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:05.786558    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:22:05.789049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:05.786673    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:22:05.789049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:05.786705    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:22:05.834713 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:05.821096    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:06.296908 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:06.296856    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:06.298571 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:06.298548    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:22:06.389309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:06.389282    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:22:06.415310 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:06.415263    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:22:06.483114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:06.476407    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:22:06.483114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:06.476508    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:22:06.594430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:06.594328    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:22:12.454612 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.454576    4441 generic.go:356] "Generic (PLEG): container finished" podID="71d7723e-9a7d-4858-8093-4b13593809db" containerID="e556283aabfe42c525e73a2e1fde7c897cbaa541a5c71f0adcdb2197190e53e9" exitCode=0
Oct 30 00:22:12.455143 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.454645    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" event={"ID":"71d7723e-9a7d-4858-8093-4b13593809db","Type":"ContainerDied","Data":"e556283aabfe42c525e73a2e1fde7c897cbaa541a5c71f0adcdb2197190e53e9"}
Oct 30 00:22:12.455143 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.455129    4441 scope.go:122] "RemoveContainer" containerID="e556283aabfe42c525e73a2e1fde7c897cbaa541a5c71f0adcdb2197190e53e9"
Oct 30 00:22:12.457066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.456993    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:22:12.457184 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.457080    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:22:12.457184 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.457118    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:22:12.585923 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.584171    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"]
Oct 30 00:22:12.585923 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.585370    4441 cpu_manager.go:414] "containerMap: removing container" podUID="13255825-7146-4789-9262-3cb401f87bd3" containerName="payload-pre-processing"
Oct 30 00:22:12.585923 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.585392    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13255825-7146-4789-9262-3cb401f87bd3" containerName="payload-pre-processing"
Oct 30 00:22:12.585923 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.585416    4441 cpu_manager.go:414] "containerMap: removing container" podUID="94a58ad1-9196-49a3-a61a-a3e7ebb5cc01" containerName="payload-processing"
Oct 30 00:22:12.585923 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.585423    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a58ad1-9196-49a3-a61a-a3e7ebb5cc01" containerName="payload-processing"
Oct 30 00:22:12.585923 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.585621    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="13255825-7146-4789-9262-3cb401f87bd3" containerName="payload-pre-processing"
Oct 30 00:22:12.585923 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.585636    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a58ad1-9196-49a3-a61a-a3e7ebb5cc01" containerName="payload-processing"
Oct 30 00:22:12.594414 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.593872    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:12.600795 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.600766    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"]
Oct 30 00:22:12.638320 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.632577    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f52ea1-serving-cert\""
Oct 30 00:22:12.709238 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.709206    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:12.709432 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.709275    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:12.709432 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.709365    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbmzk\" (UniqueName: \"kubernetes.io/projected/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-kube-api-access-gbmzk\") pod \"maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:12.709572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.709452    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-metrics-certs\") pod \"maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:12.814080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.811564    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbmzk\" (UniqueName: \"kubernetes.io/projected/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-kube-api-access-gbmzk\") pod \"maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:12.814080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.811691    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-metrics-certs\") pod \"maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:12.814080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.812017    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:12.814080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.812137    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:12.814080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.813057    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:12.817292 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.817265    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-metrics-certs\") pod \"maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:12.817475 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.817446    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:12.867427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.867329    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbmzk\" (UniqueName: \"kubernetes.io/projected/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-kube-api-access-gbmzk\") pod \"maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:12.967283 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:12.967001    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:13.299671 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:13.299636    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"]
Oct 30 00:22:13.299789 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:22:13.299677    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c61570_5301_4408_bbf4_6f59bb6d4dc9.slice/crio-1ce0e1696ac361d2de3a0621a375ea03abb77d57c5a6f4df4b697e83580ea99e WatchSource:0}: Error finding container 1ce0e1696ac361d2de3a0621a375ea03abb77d57c5a6f4df4b697e83580ea99e: Status 404 returned error can't find the container with id 1ce0e1696ac361d2de3a0621a375ea03abb77d57c5a6f4df4b697e83580ea99e
Oct 30 00:22:13.469444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:13.469399    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" event={"ID":"71d7723e-9a7d-4858-8093-4b13593809db","Type":"ContainerStarted","Data":"0a2f21822a04a80ef3e54e47bf9bdab5f353b4317c81226ca89e8e203e5eb6b1"}
Oct 30 00:22:13.472111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:13.472081    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc" event={"ID":"b1c61570-5301-4408-bbf4-6f59bb6d4dc9","Type":"ContainerStarted","Data":"1ce0e1696ac361d2de3a0621a375ea03abb77d57c5a6f4df4b697e83580ea99e"}
Oct 30 00:22:14.183523 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:14.183404    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:22:14.183703 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:14.183582    4441 patch_prober.go:29] interesting pod/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 00:22:14.183703 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:14.183637    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" podUID="71d7723e-9a7d-4858-8093-4b13593809db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 00:22:14.506180 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:14.506135    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc" event={"ID":"b1c61570-5301-4408-bbf4-6f59bb6d4dc9","Type":"ContainerStarted","Data":"cd0bd65dae0f02717dd5fd1573fe25f02c64bd1636c506983f733afa0b93017d"}
Oct 30 00:22:14.506682 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:14.506217    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:14.561241 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:14.551467    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc" podStartSLOduration=2.193916592 podStartE2EDuration="2.551446271s" totalImagesPullingTime="357.529679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:22:14.550575923 +0000 UTC m=+1044.075869801" watchObservedRunningTime="2026-10-30 00:22:14.551446271 +0000 UTC m=+1044.076740149"
Oct 30 00:22:15.183275 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:15.183237    4441 patch_prober.go:29] interesting pod/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 00:22:15.183475 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:15.183291    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" podUID="71d7723e-9a7d-4858-8093-4b13593809db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 00:22:16.183212 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:16.183177    4441 patch_prober.go:29] interesting pod/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 00:22:16.183786 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:16.183752    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" podUID="71d7723e-9a7d-4858-8093-4b13593809db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 00:22:17.183444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:17.183407    4441 patch_prober.go:29] interesting pod/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 00:22:17.183843 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:17.183476    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" podUID="71d7723e-9a7d-4858-8093-4b13593809db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 00:22:18.183324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:18.183289    4441 patch_prober.go:29] interesting pod/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 00:22:18.183529 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:18.183339    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" podUID="71d7723e-9a7d-4858-8093-4b13593809db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 00:22:18.965294 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:18.965254    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"]
Oct 30 00:22:18.970282 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:18.970255    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:18.988906 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:18.988876    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"]
Oct 30 00:22:18.997366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:18.995562    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c34304-serving-cert\""
Oct 30 00:22:19.018862 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.018813    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:19.019213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.018941    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x89sl\" (UniqueName: \"kubernetes.io/projected/bee7d813-eb28-46ed-b93c-f5422243a68a-kube-api-access-x89sl\") pod \"maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:19.019213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.018996    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee7d813-eb28-46ed-b93c-f5422243a68a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:19.019213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.019146    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-metrics-certs\") pod \"maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:19.121251 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.121214    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:19.121448 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.121318    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89sl\" (UniqueName: \"kubernetes.io/projected/bee7d813-eb28-46ed-b93c-f5422243a68a-kube-api-access-x89sl\") pod \"maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:19.121448 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.121356    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee7d813-eb28-46ed-b93c-f5422243a68a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:19.121580 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.121456    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-metrics-certs\") pod \"maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:19.122246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.122221    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee7d813-eb28-46ed-b93c-f5422243a68a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:19.123915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.123890    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:19.123999 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.123918    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-metrics-certs\") pod \"maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:19.143564 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.143536    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89sl\" (UniqueName: \"kubernetes.io/projected/bee7d813-eb28-46ed-b93c-f5422243a68a-kube-api-access-x89sl\") pod \"maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:19.188160 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.188130    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:22:19.188629 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.188606    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:22:19.189385 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.189364    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:22:19.289896 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.284956    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:19.667218 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:19.667189    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"]
Oct 30 00:22:19.667960 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:22:19.667932    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee7d813_eb28_46ed_b93c_f5422243a68a.slice/crio-6cc1d2e2004032eb15e004b619d6eb103e7370fd75b90d18a7b4694622843c88 WatchSource:0}: Error finding container 6cc1d2e2004032eb15e004b619d6eb103e7370fd75b90d18a7b4694622843c88: Status 404 returned error can't find the container with id 6cc1d2e2004032eb15e004b619d6eb103e7370fd75b90d18a7b4694622843c88
Oct 30 00:22:20.522080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:20.522011    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:22:20.599515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:20.599479    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz" event={"ID":"bee7d813-eb28-46ed-b93c-f5422243a68a","Type":"ContainerStarted","Data":"ec480bededaebdf53698a1f068604353dfb76daef31b9f68553077162ba8c9a8"}
Oct 30 00:22:20.599696 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:20.599546    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz" event={"ID":"bee7d813-eb28-46ed-b93c-f5422243a68a","Type":"ContainerStarted","Data":"6cc1d2e2004032eb15e004b619d6eb103e7370fd75b90d18a7b4694622843c88"}
Oct 30 00:22:20.599696 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:20.599579    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:20.644016 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:20.643967    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz" podStartSLOduration=2.3281740060000002 podStartE2EDuration="2.643952241s" totalImagesPullingTime="315.778235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:22:20.639558212 +0000 UTC m=+1050.164852091" watchObservedRunningTime="2026-10-30 00:22:20.643952241 +0000 UTC m=+1050.169246123"
Oct 30 00:22:26.613000 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:26.612974    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:22:26.947389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:26.947285    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"]
Oct 30 00:22:31.947841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:31.947781    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p" podUID="eccb4296-5590-42ff-9ea8-870c68993622" containerName="payload-processing" containerID="cri-o://db3d62d66eb15206939106b53205411498d79790ecaeb70f5eae848f17fd8517" gracePeriod=25
Oct 30 00:22:32.286828 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.286801    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"
Oct 30 00:22:32.372009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.371978    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eccb4296-5590-42ff-9ea8-870c68993622-plugins-config-volume" pod "eccb4296-5590-42ff-9ea8-870c68993622" (UID: "eccb4296-5590-42ff-9ea8-870c68993622"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:22:32.372585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.371599    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eccb4296-5590-42ff-9ea8-870c68993622-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccb4296-5590-42ff-9ea8-870c68993622-plugins-config-volume\") pod \"eccb4296-5590-42ff-9ea8-870c68993622\" (UID: \"eccb4296-5590-42ff-9ea8-870c68993622\") "
Oct 30 00:22:32.373113 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.373085    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eccb4296-5590-42ff-9ea8-870c68993622-kube-api-access-qnmmw\" (UniqueName: \"kubernetes.io/projected/eccb4296-5590-42ff-9ea8-870c68993622-kube-api-access-qnmmw\") pod \"eccb4296-5590-42ff-9ea8-870c68993622\" (UID: \"eccb4296-5590-42ff-9ea8-870c68993622\") "
Oct 30 00:22:32.375839 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.373796    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccb4296-5590-42ff-9ea8-870c68993622-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:22:32.376166 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.375942    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eccb4296-5590-42ff-9ea8-870c68993622-kube-api-access-qnmmw" pod "eccb4296-5590-42ff-9ea8-870c68993622" (UID: "eccb4296-5590-42ff-9ea8-870c68993622"). InnerVolumeSpecName "kube-api-access-qnmmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:22:32.475731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.475644    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnmmw\" (UniqueName: \"kubernetes.io/projected/eccb4296-5590-42ff-9ea8-870c68993622-kube-api-access-qnmmw\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:22:32.759200 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.759159    4441 generic.go:356] "Generic (PLEG): container finished" podID="eccb4296-5590-42ff-9ea8-870c68993622" containerID="db3d62d66eb15206939106b53205411498d79790ecaeb70f5eae848f17fd8517" exitCode=0
Oct 30 00:22:32.759379 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.759270    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"
Oct 30 00:22:32.759379 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.759320    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p" event={"ID":"eccb4296-5590-42ff-9ea8-870c68993622","Type":"ContainerDied","Data":"db3d62d66eb15206939106b53205411498d79790ecaeb70f5eae848f17fd8517"}
Oct 30 00:22:32.759510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.759412    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p" event={"ID":"eccb4296-5590-42ff-9ea8-870c68993622","Type":"ContainerDied","Data":"54a0dfd568b6f7ec0aa9010203f957b937e4f74d20f524c37b61696e386c51ad"}
Oct 30 00:22:32.759510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.759455    4441 scope.go:122] "RemoveContainer" containerID="db3d62d66eb15206939106b53205411498d79790ecaeb70f5eae848f17fd8517"
Oct 30 00:22:32.773616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.773593    4441 scope.go:122] "RemoveContainer" containerID="db3d62d66eb15206939106b53205411498d79790ecaeb70f5eae848f17fd8517"
Oct 30 00:22:32.773927 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:22:32.773902    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3d62d66eb15206939106b53205411498d79790ecaeb70f5eae848f17fd8517\": container with ID starting with db3d62d66eb15206939106b53205411498d79790ecaeb70f5eae848f17fd8517 not found: ID does not exist" containerID="db3d62d66eb15206939106b53205411498d79790ecaeb70f5eae848f17fd8517"
Oct 30 00:22:32.774054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.773941    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3d62d66eb15206939106b53205411498d79790ecaeb70f5eae848f17fd8517"} err="failed to get container status \"db3d62d66eb15206939106b53205411498d79790ecaeb70f5eae848f17fd8517\": rpc error: code = NotFound desc = could not find container \"db3d62d66eb15206939106b53205411498d79790ecaeb70f5eae848f17fd8517\": container with ID starting with db3d62d66eb15206939106b53205411498d79790ecaeb70f5eae848f17fd8517 not found: ID does not exist"
Oct 30 00:22:32.810564 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.810535    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"]
Oct 30 00:22:32.821985 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:32.821954    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-xwm7p"]
Oct 30 00:22:33.904712 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:22:33.904675    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:22:34.612172 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:34.612136    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eccb4296-5590-42ff-9ea8-870c68993622" path="/var/lib/kubelet/pods/eccb4296-5590-42ff-9ea8-870c68993622/volumes"
Oct 30 00:22:38.263340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:38.263299    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:22:59.162520 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.162488    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"]
Oct 30 00:22:59.163249 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.163231    4441 cpu_manager.go:414] "containerMap: removing container" podUID="eccb4296-5590-42ff-9ea8-870c68993622" containerName="payload-processing"
Oct 30 00:22:59.163321 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.163251    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eccb4296-5590-42ff-9ea8-870c68993622" containerName="payload-processing"
Oct 30 00:22:59.163424 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.163413    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="eccb4296-5590-42ff-9ea8-870c68993622" containerName="payload-processing"
Oct 30 00:22:59.166863 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.166843    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.270684 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.270660    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f52ea1-openshift-default-dockercfg-6zrs8\""
Oct 30 00:22:59.279440 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.279415    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx4l9\" (UniqueName: \"kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-kube-api-access-kx4l9\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.279558 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.279484    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istiod-ca-cert\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.279597 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.279569    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-envoy\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.279693 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.279678    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-podinfo\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.279742 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.279709    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.279786 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.279759    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-certs\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.279832 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.279816    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-credential-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.279909 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.279895    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-data\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.279956 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.279945    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-token\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.381057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.380999    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istiod-ca-cert\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.381268 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.381100    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-envoy\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.381268 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.381210    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-podinfo\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.381268 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.381252    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.381399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.381292    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-certs\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.381399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.381337    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-credential-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.381683 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.381644    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-certs\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.381788 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.381707    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-credential-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.381852 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.381809    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istiod-ca-cert\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.381852 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.381834    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-data\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.381961 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.381921    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-token\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.382020 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.381966    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.382020 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.381994    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx4l9\" (UniqueName: \"kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-kube-api-access-kx4l9\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.382191 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.382110    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-data\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.383605 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.383571    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-envoy\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.383866 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.383846    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-podinfo\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.404458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.404426    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-token\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.404801 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.404775    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx4l9\" (UniqueName: \"kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-kube-api-access-kx4l9\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:22:59.487330 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:22:59.484410    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:23:03.901981 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:23:03.901936    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:23:05.979888 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:05.979843    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"]
Oct 30 00:23:06.356008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:06.355976    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"]
Oct 30 00:23:06.359239 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:23:06.359211    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936b1610_9e63_4eb6_95c3_8d2e10d307f7.slice/crio-79eabcad0030ed7ab971c9b28673d2556f6e901ba020bbd131d204dfc84369a0 WatchSource:0}: Error finding container 79eabcad0030ed7ab971c9b28673d2556f6e901ba020bbd131d204dfc84369a0: Status 404 returned error can't find the container with id 79eabcad0030ed7ab971c9b28673d2556f6e901ba020bbd131d204dfc84369a0
Oct 30 00:23:06.360422 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:06.360382    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:23:06.360610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:06.360583    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:23:06.361104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:06.361079    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:23:07.160169 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:07.153181    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l" event={"ID":"936b1610-9e63-4eb6-95c3-8d2e10d307f7","Type":"ContainerStarted","Data":"30176cca34d99e80bc6f70a5c2dbdefd480f298891c0ef54f4724549af70d15a"}
Oct 30 00:23:07.160169 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:07.153264    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l" event={"ID":"936b1610-9e63-4eb6-95c3-8d2e10d307f7","Type":"ContainerStarted","Data":"79eabcad0030ed7ab971c9b28673d2556f6e901ba020bbd131d204dfc84369a0"}
Oct 30 00:23:07.226188 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:07.226135    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l" podStartSLOduration=8.226116434 podStartE2EDuration="8.226116434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:23:07.211347181 +0000 UTC m=+1096.736641056" watchObservedRunningTime="2026-10-30 00:23:07.226116434 +0000 UTC m=+1096.751410311"
Oct 30 00:23:07.485289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:07.485255    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:23:07.486950 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:07.486924    4441 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 00:23:07.487104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:07.486987    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l" podUID="936b1610-9e63-4eb6-95c3-8d2e10d307f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 00:23:08.485603 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:08.485568    4441 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 00:23:08.485997 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:08.485618    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l" podUID="936b1610-9e63-4eb6-95c3-8d2e10d307f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 00:23:09.485335 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:09.485294    4441 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 00:23:09.485535 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:09.485356    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l" podUID="936b1610-9e63-4eb6-95c3-8d2e10d307f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 00:23:10.529663 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:10.529627    4441 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 00:23:10.582062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:10.580226    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l" podUID="936b1610-9e63-4eb6-95c3-8d2e10d307f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 00:23:11.486355 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:11.486317    4441 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 00:23:11.486532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:11.486384    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l" podUID="936b1610-9e63-4eb6-95c3-8d2e10d307f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 00:23:12.485437 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:12.485400    4441 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 00:23:12.485825 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:12.485459    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l" podUID="936b1610-9e63-4eb6-95c3-8d2e10d307f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 00:23:13.489236 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:13.489205    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:23:13.489750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:13.489731    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:23:13.490485 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:13.490468    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:23:15.610652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:15.610614    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"]
Oct 30 00:23:15.645019 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:15.644989    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"]
Oct 30 00:23:16.402014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:16.401131    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"]
Oct 30 00:23:18.110139 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:18.110103    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"]
Oct 30 00:23:18.634554 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:18.634517    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:23:20.611177 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:20.611074    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk" podUID="58f56459-1511-43bf-83bb-3fa7d5efeb9a" containerName="payload-processing" containerID="cri-o://d3afb2dba2452d2e4b75606fdbad53a3efbc41665fa3f129a3eabbf71442f52c" gracePeriod=25
Oct 30 00:23:20.645872 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:20.645828    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8" podUID="b3de2b1e-69bd-4475-a45c-1904f032657c" containerName="payload-pre-processing" containerID="cri-o://2e130ddd96a7537df959184a2c3ff2512b146ccf255e197d8c8e583f70bedef6" gracePeriod=25
Oct 30 00:23:20.962106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:20.962073    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"
Oct 30 00:23:20.992950 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:20.992918    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"
Oct 30 00:23:21.085013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.078738    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f56459-1511-43bf-83bb-3fa7d5efeb9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f56459-1511-43bf-83bb-3fa7d5efeb9a-plugins-config-volume\") pod \"58f56459-1511-43bf-83bb-3fa7d5efeb9a\" (UID: \"58f56459-1511-43bf-83bb-3fa7d5efeb9a\") "
Oct 30 00:23:21.085013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.078809    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3de2b1e-69bd-4475-a45c-1904f032657c-kube-api-access-xjq4n\" (UniqueName: \"kubernetes.io/projected/b3de2b1e-69bd-4475-a45c-1904f032657c-kube-api-access-xjq4n\") pod \"b3de2b1e-69bd-4475-a45c-1904f032657c\" (UID: \"b3de2b1e-69bd-4475-a45c-1904f032657c\") "
Oct 30 00:23:21.085013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.078874    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f56459-1511-43bf-83bb-3fa7d5efeb9a-kube-api-access-x9wbr\" (UniqueName: \"kubernetes.io/projected/58f56459-1511-43bf-83bb-3fa7d5efeb9a-kube-api-access-x9wbr\") pod \"58f56459-1511-43bf-83bb-3fa7d5efeb9a\" (UID: \"58f56459-1511-43bf-83bb-3fa7d5efeb9a\") "
Oct 30 00:23:21.085838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.085793    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f56459-1511-43bf-83bb-3fa7d5efeb9a-plugins-config-volume" pod "58f56459-1511-43bf-83bb-3fa7d5efeb9a" (UID: "58f56459-1511-43bf-83bb-3fa7d5efeb9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:23:21.086188 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.086161    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3de2b1e-69bd-4475-a45c-1904f032657c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3de2b1e-69bd-4475-a45c-1904f032657c-plugins-config-volume\") pod \"b3de2b1e-69bd-4475-a45c-1904f032657c\" (UID: \"b3de2b1e-69bd-4475-a45c-1904f032657c\") "
Oct 30 00:23:21.086761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.086733    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3de2b1e-69bd-4475-a45c-1904f032657c-plugins-config-volume" pod "b3de2b1e-69bd-4475-a45c-1904f032657c" (UID: "b3de2b1e-69bd-4475-a45c-1904f032657c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:23:21.087012 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.086955    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3de2b1e-69bd-4475-a45c-1904f032657c-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:21.087012 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.086981    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f56459-1511-43bf-83bb-3fa7d5efeb9a-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:21.088360 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.088327    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3de2b1e-69bd-4475-a45c-1904f032657c-kube-api-access-xjq4n" pod "b3de2b1e-69bd-4475-a45c-1904f032657c" (UID: "b3de2b1e-69bd-4475-a45c-1904f032657c"). InnerVolumeSpecName "kube-api-access-xjq4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:23:21.088630 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.088612    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f56459-1511-43bf-83bb-3fa7d5efeb9a-kube-api-access-x9wbr" pod "58f56459-1511-43bf-83bb-3fa7d5efeb9a" (UID: "58f56459-1511-43bf-83bb-3fa7d5efeb9a"). InnerVolumeSpecName "kube-api-access-x9wbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:23:21.189391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.189302    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjq4n\" (UniqueName: \"kubernetes.io/projected/b3de2b1e-69bd-4475-a45c-1904f032657c-kube-api-access-xjq4n\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:21.189391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.189350    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9wbr\" (UniqueName: \"kubernetes.io/projected/58f56459-1511-43bf-83bb-3fa7d5efeb9a-kube-api-access-x9wbr\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:21.383217 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.383183    4441 generic.go:356] "Generic (PLEG): container finished" podID="b3de2b1e-69bd-4475-a45c-1904f032657c" containerID="2e130ddd96a7537df959184a2c3ff2512b146ccf255e197d8c8e583f70bedef6" exitCode=0
Oct 30 00:23:21.383374 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.383232    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"
Oct 30 00:23:21.383374 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.383260    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8" event={"ID":"b3de2b1e-69bd-4475-a45c-1904f032657c","Type":"ContainerDied","Data":"2e130ddd96a7537df959184a2c3ff2512b146ccf255e197d8c8e583f70bedef6"}
Oct 30 00:23:21.383374 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.383321    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8" event={"ID":"b3de2b1e-69bd-4475-a45c-1904f032657c","Type":"ContainerDied","Data":"b91e27fa5812f6b37d280e53476d1aef5a2ad2b3d8935c3484212e8f7139786b"}
Oct 30 00:23:21.383374 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.383362    4441 scope.go:122] "RemoveContainer" containerID="2e130ddd96a7537df959184a2c3ff2512b146ccf255e197d8c8e583f70bedef6"
Oct 30 00:23:21.385138 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.385112    4441 generic.go:356] "Generic (PLEG): container finished" podID="58f56459-1511-43bf-83bb-3fa7d5efeb9a" containerID="d3afb2dba2452d2e4b75606fdbad53a3efbc41665fa3f129a3eabbf71442f52c" exitCode=0
Oct 30 00:23:21.385256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.385178    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"
Oct 30 00:23:21.385256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.385194    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk" event={"ID":"58f56459-1511-43bf-83bb-3fa7d5efeb9a","Type":"ContainerDied","Data":"d3afb2dba2452d2e4b75606fdbad53a3efbc41665fa3f129a3eabbf71442f52c"}
Oct 30 00:23:21.385346 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.385257    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk" event={"ID":"58f56459-1511-43bf-83bb-3fa7d5efeb9a","Type":"ContainerDied","Data":"8a47b9002be265b7c721ea0cda35cac6833dc99f63b8f048bf10bb7c8eafd139"}
Oct 30 00:23:21.396885 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.396769    4441 scope.go:122] "RemoveContainer" containerID="2e130ddd96a7537df959184a2c3ff2512b146ccf255e197d8c8e583f70bedef6"
Oct 30 00:23:21.397272 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:23:21.397130    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e130ddd96a7537df959184a2c3ff2512b146ccf255e197d8c8e583f70bedef6\": container with ID starting with 2e130ddd96a7537df959184a2c3ff2512b146ccf255e197d8c8e583f70bedef6 not found: ID does not exist" containerID="2e130ddd96a7537df959184a2c3ff2512b146ccf255e197d8c8e583f70bedef6"
Oct 30 00:23:21.397272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.397164    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e130ddd96a7537df959184a2c3ff2512b146ccf255e197d8c8e583f70bedef6"} err="failed to get container status \"2e130ddd96a7537df959184a2c3ff2512b146ccf255e197d8c8e583f70bedef6\": rpc error: code = NotFound desc = could not find container \"2e130ddd96a7537df959184a2c3ff2512b146ccf255e197d8c8e583f70bedef6\": container with ID starting with 2e130ddd96a7537df959184a2c3ff2512b146ccf255e197d8c8e583f70bedef6 not found: ID does not exist"
Oct 30 00:23:21.397272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.397187    4441 scope.go:122] "RemoveContainer" containerID="d3afb2dba2452d2e4b75606fdbad53a3efbc41665fa3f129a3eabbf71442f52c"
Oct 30 00:23:21.401460 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.401437    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57" podUID="79e0cda7-5bcb-4c13-98cc-92b7a9144b31" containerName="payload-processing" containerID="cri-o://99201956787ca385626119b03183d7b0a477fda613a39b02022acea4501ee0f2" gracePeriod=25
Oct 30 00:23:21.413014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.412990    4441 scope.go:122] "RemoveContainer" containerID="d3afb2dba2452d2e4b75606fdbad53a3efbc41665fa3f129a3eabbf71442f52c"
Oct 30 00:23:21.413608 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:23:21.413586    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3afb2dba2452d2e4b75606fdbad53a3efbc41665fa3f129a3eabbf71442f52c\": container with ID starting with d3afb2dba2452d2e4b75606fdbad53a3efbc41665fa3f129a3eabbf71442f52c not found: ID does not exist" containerID="d3afb2dba2452d2e4b75606fdbad53a3efbc41665fa3f129a3eabbf71442f52c"
Oct 30 00:23:21.413733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.413615    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3afb2dba2452d2e4b75606fdbad53a3efbc41665fa3f129a3eabbf71442f52c"} err="failed to get container status \"d3afb2dba2452d2e4b75606fdbad53a3efbc41665fa3f129a3eabbf71442f52c\": rpc error: code = NotFound desc = could not find container \"d3afb2dba2452d2e4b75606fdbad53a3efbc41665fa3f129a3eabbf71442f52c\": container with ID starting with d3afb2dba2452d2e4b75606fdbad53a3efbc41665fa3f129a3eabbf71442f52c not found: ID does not exist"
Oct 30 00:23:21.436148 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.436123    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"]
Oct 30 00:23:21.440145 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.440094    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-461609-7bdf6c54fd-8tvlk"]
Oct 30 00:23:21.471058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.470994    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"]
Oct 30 00:23:21.474462 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.474442    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-461609-5c9d9ff695-zfdt8"]
Oct 30 00:23:21.729251 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.729224    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"
Oct 30 00:23:21.906704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.906675    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-plugins-config-volume\") pod \"79e0cda7-5bcb-4c13-98cc-92b7a9144b31\" (UID: \"79e0cda7-5bcb-4c13-98cc-92b7a9144b31\") "
Oct 30 00:23:21.906871 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.906857    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-kube-api-access-vgdrm\" (UniqueName: \"kubernetes.io/projected/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-kube-api-access-vgdrm\") pod \"79e0cda7-5bcb-4c13-98cc-92b7a9144b31\" (UID: \"79e0cda7-5bcb-4c13-98cc-92b7a9144b31\") "
Oct 30 00:23:21.907131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.907106    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-plugins-config-volume" pod "79e0cda7-5bcb-4c13-98cc-92b7a9144b31" (UID: "79e0cda7-5bcb-4c13-98cc-92b7a9144b31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:23:21.907539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.907518    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:21.908937 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:21.908914    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-kube-api-access-vgdrm" pod "79e0cda7-5bcb-4c13-98cc-92b7a9144b31" (UID: "79e0cda7-5bcb-4c13-98cc-92b7a9144b31"). InnerVolumeSpecName "kube-api-access-vgdrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:23:22.009351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:22.009319    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgdrm\" (UniqueName: \"kubernetes.io/projected/79e0cda7-5bcb-4c13-98cc-92b7a9144b31-kube-api-access-vgdrm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:22.397294 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:22.397202    4441 generic.go:356] "Generic (PLEG): container finished" podID="79e0cda7-5bcb-4c13-98cc-92b7a9144b31" containerID="99201956787ca385626119b03183d7b0a477fda613a39b02022acea4501ee0f2" exitCode=0
Oct 30 00:23:22.397451 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:22.397408    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57" event={"ID":"79e0cda7-5bcb-4c13-98cc-92b7a9144b31","Type":"ContainerDied","Data":"99201956787ca385626119b03183d7b0a477fda613a39b02022acea4501ee0f2"}
Oct 30 00:23:22.397498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:22.397458    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57" event={"ID":"79e0cda7-5bcb-4c13-98cc-92b7a9144b31","Type":"ContainerDied","Data":"fa2cf6a862ccf71f549e3f25735a4265e14044c1f0c38fee51a4e1c74d399f17"}
Oct 30 00:23:22.397498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:22.397484    4441 scope.go:122] "RemoveContainer" containerID="99201956787ca385626119b03183d7b0a477fda613a39b02022acea4501ee0f2"
Oct 30 00:23:22.397587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:22.397574    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"
Oct 30 00:23:22.411685 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:22.411648    4441 scope.go:122] "RemoveContainer" containerID="99201956787ca385626119b03183d7b0a477fda613a39b02022acea4501ee0f2"
Oct 30 00:23:22.411910 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:23:22.411887    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99201956787ca385626119b03183d7b0a477fda613a39b02022acea4501ee0f2\": container with ID starting with 99201956787ca385626119b03183d7b0a477fda613a39b02022acea4501ee0f2 not found: ID does not exist" containerID="99201956787ca385626119b03183d7b0a477fda613a39b02022acea4501ee0f2"
Oct 30 00:23:22.411982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:22.411919    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99201956787ca385626119b03183d7b0a477fda613a39b02022acea4501ee0f2"} err="failed to get container status \"99201956787ca385626119b03183d7b0a477fda613a39b02022acea4501ee0f2\": rpc error: code = NotFound desc = could not find container \"99201956787ca385626119b03183d7b0a477fda613a39b02022acea4501ee0f2\": container with ID starting with 99201956787ca385626119b03183d7b0a477fda613a39b02022acea4501ee0f2 not found: ID does not exist"
Oct 30 00:23:22.447856 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:22.447829    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"]
Oct 30 00:23:22.451691 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:22.451668    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7417dc-7d779c855b-9ks57"]
Oct 30 00:23:22.610936 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:22.610910    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f56459-1511-43bf-83bb-3fa7d5efeb9a" path="/var/lib/kubelet/pods/58f56459-1511-43bf-83bb-3fa7d5efeb9a/volumes"
Oct 30 00:23:22.611393 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:22.611377    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e0cda7-5bcb-4c13-98cc-92b7a9144b31" path="/var/lib/kubelet/pods/79e0cda7-5bcb-4c13-98cc-92b7a9144b31/volumes"
Oct 30 00:23:22.611827 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:22.611810    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3de2b1e-69bd-4475-a45c-1904f032657c" path="/var/lib/kubelet/pods/b3de2b1e-69bd-4475-a45c-1904f032657c/volumes"
Oct 30 00:23:23.113601 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:23.113524    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h" podUID="8660a915-4fe9-4b55-bde2-30d2222da065" containerName="payload-processing" containerID="cri-o://d62ef154bd0b9b95b4adb3e1b134936c5e10d7ee3f4b8507e57043525966788d" gracePeriod=25
Oct 30 00:23:23.443616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:23.432854    4441 generic.go:356] "Generic (PLEG): container finished" podID="8660a915-4fe9-4b55-bde2-30d2222da065" containerID="d62ef154bd0b9b95b4adb3e1b134936c5e10d7ee3f4b8507e57043525966788d" exitCode=0
Oct 30 00:23:23.443616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:23.432959    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h" event={"ID":"8660a915-4fe9-4b55-bde2-30d2222da065","Type":"ContainerDied","Data":"d62ef154bd0b9b95b4adb3e1b134936c5e10d7ee3f4b8507e57043525966788d"}
Oct 30 00:23:23.463783 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:23.463758    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"
Oct 30 00:23:23.635821 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:23.635788    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8660a915-4fe9-4b55-bde2-30d2222da065-kube-api-access-tcv4g\" (UniqueName: \"kubernetes.io/projected/8660a915-4fe9-4b55-bde2-30d2222da065-kube-api-access-tcv4g\") pod \"8660a915-4fe9-4b55-bde2-30d2222da065\" (UID: \"8660a915-4fe9-4b55-bde2-30d2222da065\") "
Oct 30 00:23:23.635983 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:23.635973    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8660a915-4fe9-4b55-bde2-30d2222da065-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8660a915-4fe9-4b55-bde2-30d2222da065-plugins-config-volume\") pod \"8660a915-4fe9-4b55-bde2-30d2222da065\" (UID: \"8660a915-4fe9-4b55-bde2-30d2222da065\") "
Oct 30 00:23:23.636374 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:23.636338    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8660a915-4fe9-4b55-bde2-30d2222da065-plugins-config-volume" pod "8660a915-4fe9-4b55-bde2-30d2222da065" (UID: "8660a915-4fe9-4b55-bde2-30d2222da065"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:23:23.637145 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:23.637119    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8660a915-4fe9-4b55-bde2-30d2222da065-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:23.638345 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:23.638322    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8660a915-4fe9-4b55-bde2-30d2222da065-kube-api-access-tcv4g" pod "8660a915-4fe9-4b55-bde2-30d2222da065" (UID: "8660a915-4fe9-4b55-bde2-30d2222da065"). InnerVolumeSpecName "kube-api-access-tcv4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:23:23.738512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:23.738482    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcv4g\" (UniqueName: \"kubernetes.io/projected/8660a915-4fe9-4b55-bde2-30d2222da065-kube-api-access-tcv4g\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:24.446095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:24.446062    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"
Oct 30 00:23:24.446095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:24.446068    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h" event={"ID":"8660a915-4fe9-4b55-bde2-30d2222da065","Type":"ContainerDied","Data":"e28e37857cc7755f93312e68847b80953b9fa002dd52cdb9819cbf54e460e7d8"}
Oct 30 00:23:24.446593 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:24.446156    4441 scope.go:122] "RemoveContainer" containerID="d62ef154bd0b9b95b4adb3e1b134936c5e10d7ee3f4b8507e57043525966788d"
Oct 30 00:23:24.496686 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:24.496657    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"]
Oct 30 00:23:24.509223 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:24.509196    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e2a011-7fd544f554-gtb2h"]
Oct 30 00:23:24.611710 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:24.611684    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8660a915-4fe9-4b55-bde2-30d2222da065" path="/var/lib/kubelet/pods/8660a915-4fe9-4b55-bde2-30d2222da065/volumes"
Oct 30 00:23:33.500800 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.500764    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"]
Oct 30 00:23:33.501506 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.501488    4441 cpu_manager.go:414] "containerMap: removing container" podUID="58f56459-1511-43bf-83bb-3fa7d5efeb9a" containerName="payload-processing"
Oct 30 00:23:33.501556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.501509    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f56459-1511-43bf-83bb-3fa7d5efeb9a" containerName="payload-processing"
Oct 30 00:23:33.501556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.501545    4441 cpu_manager.go:414] "containerMap: removing container" podUID="79e0cda7-5bcb-4c13-98cc-92b7a9144b31" containerName="payload-processing"
Oct 30 00:23:33.501556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.501550    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e0cda7-5bcb-4c13-98cc-92b7a9144b31" containerName="payload-processing"
Oct 30 00:23:33.501647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.501566    4441 cpu_manager.go:414] "containerMap: removing container" podUID="8660a915-4fe9-4b55-bde2-30d2222da065" containerName="payload-processing"
Oct 30 00:23:33.501647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.501571    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8660a915-4fe9-4b55-bde2-30d2222da065" containerName="payload-processing"
Oct 30 00:23:33.501647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.501583    4441 cpu_manager.go:414] "containerMap: removing container" podUID="b3de2b1e-69bd-4475-a45c-1904f032657c" containerName="payload-pre-processing"
Oct 30 00:23:33.501647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.501588    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3de2b1e-69bd-4475-a45c-1904f032657c" containerName="payload-pre-processing"
Oct 30 00:23:33.501764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.501685    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f56459-1511-43bf-83bb-3fa7d5efeb9a" containerName="payload-processing"
Oct 30 00:23:33.501764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.501697    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e0cda7-5bcb-4c13-98cc-92b7a9144b31" containerName="payload-processing"
Oct 30 00:23:33.501764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.501705    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8660a915-4fe9-4b55-bde2-30d2222da065" containerName="payload-processing"
Oct 30 00:23:33.501764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.501712    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3de2b1e-69bd-4475-a45c-1904f032657c" containerName="payload-pre-processing"
Oct 30 00:23:33.505198 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.505176    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"
Oct 30 00:23:33.530661 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.530636    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6a55e1\""
Oct 30 00:23:33.530661 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.530640    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6a55e1-dockercfg-lscpp\""
Oct 30 00:23:33.612049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.611057    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96327c70-a528-47d6-ac5d-d68072291d6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv\" (UID: \"96327c70-a528-47d6-ac5d-d68072291d6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"
Oct 30 00:23:33.612049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.611192    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwtbs\" (UniqueName: \"kubernetes.io/projected/96327c70-a528-47d6-ac5d-d68072291d6d-kube-api-access-wwtbs\") pod \"payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv\" (UID: \"96327c70-a528-47d6-ac5d-d68072291d6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"
Oct 30 00:23:33.630819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.622292    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"]
Oct 30 00:23:33.649987 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.649956    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"
Oct 30 00:23:33.756096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.753211    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7\" (UID: \"caaf8af0-c2b6-4544-896e-fc9cf76e0e99\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"
Oct 30 00:23:33.756096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.753362    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96327c70-a528-47d6-ac5d-d68072291d6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv\" (UID: \"96327c70-a528-47d6-ac5d-d68072291d6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"
Oct 30 00:23:33.756096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.753571    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwtbs\" (UniqueName: \"kubernetes.io/projected/96327c70-a528-47d6-ac5d-d68072291d6d-kube-api-access-wwtbs\") pod \"payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv\" (UID: \"96327c70-a528-47d6-ac5d-d68072291d6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"
Oct 30 00:23:33.756096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.753642    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9v7\" (UniqueName: \"kubernetes.io/projected/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-kube-api-access-4t9v7\") pod \"payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7\" (UID: \"caaf8af0-c2b6-4544-896e-fc9cf76e0e99\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"
Oct 30 00:23:33.756096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.755590    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96327c70-a528-47d6-ac5d-d68072291d6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv\" (UID: \"96327c70-a528-47d6-ac5d-d68072291d6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"
Oct 30 00:23:33.797537 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.785931    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwtbs\" (UniqueName: \"kubernetes.io/projected/96327c70-a528-47d6-ac5d-d68072291d6d-kube-api-access-wwtbs\") pod \"payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv\" (UID: \"96327c70-a528-47d6-ac5d-d68072291d6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"
Oct 30 00:23:33.819740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.819705    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"
Oct 30 00:23:33.854975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.854932    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9v7\" (UniqueName: \"kubernetes.io/projected/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-kube-api-access-4t9v7\") pod \"payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7\" (UID: \"caaf8af0-c2b6-4544-896e-fc9cf76e0e99\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"
Oct 30 00:23:33.855229 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.855208    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7\" (UID: \"caaf8af0-c2b6-4544-896e-fc9cf76e0e99\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"
Oct 30 00:23:33.861107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.856401    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7\" (UID: \"caaf8af0-c2b6-4544-896e-fc9cf76e0e99\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"
Oct 30 00:23:33.897090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.889084    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9v7\" (UniqueName: \"kubernetes.io/projected/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-kube-api-access-4t9v7\") pod \"payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7\" (UID: \"caaf8af0-c2b6-4544-896e-fc9cf76e0e99\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"
Oct 30 00:23:33.919475 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:23:33.919397    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:23:33.993277 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:33.993238    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"
Oct 30 00:23:35.263709 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.263671    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"]
Oct 30 00:23:35.268962 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.268937    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"
Oct 30 00:23:35.296377 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.296354    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-96027e-dockercfg-mzpcl\""
Oct 30 00:23:35.296468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.296410    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-96027e\""
Oct 30 00:23:35.409065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.400709    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8774a1-1532-46be-8af1-086a1d709c01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6\" (UID: \"ea8774a1-1532-46be-8af1-086a1d709c01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"
Oct 30 00:23:35.409065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.400799    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mbgp\" (UniqueName: \"kubernetes.io/projected/ea8774a1-1532-46be-8af1-086a1d709c01-kube-api-access-7mbgp\") pod \"payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6\" (UID: \"ea8774a1-1532-46be-8af1-086a1d709c01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"
Oct 30 00:23:35.464895 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.464862    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"]
Oct 30 00:23:35.465400 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.465365    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl" podUID="402d330d-6644-40e9-9bbe-8775bd0fadc4" containerName="maas-api" containerID="cri-o://43e63ed303a196c7b5fe2985cdde441f05efae67cfaec34661205711ac7031e1" gracePeriod=30
Oct 30 00:23:35.515591 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.515514    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8774a1-1532-46be-8af1-086a1d709c01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6\" (UID: \"ea8774a1-1532-46be-8af1-086a1d709c01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"
Oct 30 00:23:35.515757 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.515589    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mbgp\" (UniqueName: \"kubernetes.io/projected/ea8774a1-1532-46be-8af1-086a1d709c01-kube-api-access-7mbgp\") pod \"payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6\" (UID: \"ea8774a1-1532-46be-8af1-086a1d709c01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"
Oct 30 00:23:35.524643 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.522559    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8774a1-1532-46be-8af1-086a1d709c01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6\" (UID: \"ea8774a1-1532-46be-8af1-086a1d709c01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"
Oct 30 00:23:35.649858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.649780    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mbgp\" (UniqueName: \"kubernetes.io/projected/ea8774a1-1532-46be-8af1-086a1d709c01-kube-api-access-7mbgp\") pod \"payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6\" (UID: \"ea8774a1-1532-46be-8af1-086a1d709c01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"
Oct 30 00:23:35.888141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.885664    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"
Oct 30 00:23:35.986119 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:35.985264    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:23:36.035878 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.035832    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-metrics-certs\") pod \"402d330d-6644-40e9-9bbe-8775bd0fadc4\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") "
Oct 30 00:23:36.036161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.035985    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/402d330d-6644-40e9-9bbe-8775bd0fadc4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402d330d-6644-40e9-9bbe-8775bd0fadc4-openshift-service-ca\") pod \"402d330d-6644-40e9-9bbe-8775bd0fadc4\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") "
Oct 30 00:23:36.036264 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.036187    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402d330d-6644-40e9-9bbe-8775bd0fadc4-kube-api-access-drnqj\" (UniqueName: \"kubernetes.io/projected/402d330d-6644-40e9-9bbe-8775bd0fadc4-kube-api-access-drnqj\") pod \"402d330d-6644-40e9-9bbe-8775bd0fadc4\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") "
Oct 30 00:23:36.036373 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.036356    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-maas-api-tls\") pod \"402d330d-6644-40e9-9bbe-8775bd0fadc4\" (UID: \"402d330d-6644-40e9-9bbe-8775bd0fadc4\") "
Oct 30 00:23:36.036485 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.036427    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402d330d-6644-40e9-9bbe-8775bd0fadc4-openshift-service-ca" pod "402d330d-6644-40e9-9bbe-8775bd0fadc4" (UID: "402d330d-6644-40e9-9bbe-8775bd0fadc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:23:36.037227 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.037198    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402d330d-6644-40e9-9bbe-8775bd0fadc4-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:36.039278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.039232    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-maas-api-tls" pod "402d330d-6644-40e9-9bbe-8775bd0fadc4" (UID: "402d330d-6644-40e9-9bbe-8775bd0fadc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:23:36.039377 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.039305    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-metrics-certs" pod "402d330d-6644-40e9-9bbe-8775bd0fadc4" (UID: "402d330d-6644-40e9-9bbe-8775bd0fadc4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:23:36.040998 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.040975    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402d330d-6644-40e9-9bbe-8775bd0fadc4-kube-api-access-drnqj" pod "402d330d-6644-40e9-9bbe-8775bd0fadc4" (UID: "402d330d-6644-40e9-9bbe-8775bd0fadc4"). InnerVolumeSpecName "kube-api-access-drnqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:23:36.139840 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.139698    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:36.139840 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.139737    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drnqj\" (UniqueName: \"kubernetes.io/projected/402d330d-6644-40e9-9bbe-8775bd0fadc4-kube-api-access-drnqj\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:36.139840 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.139755    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402d330d-6644-40e9-9bbe-8775bd0fadc4-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:36.669144 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.669115    4441 generic.go:356] "Generic (PLEG): container finished" podID="402d330d-6644-40e9-9bbe-8775bd0fadc4" containerID="43e63ed303a196c7b5fe2985cdde441f05efae67cfaec34661205711ac7031e1" exitCode=0
Oct 30 00:23:36.669620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.669167    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"
Oct 30 00:23:36.669620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.669195    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl" event={"ID":"402d330d-6644-40e9-9bbe-8775bd0fadc4","Type":"ContainerDied","Data":"43e63ed303a196c7b5fe2985cdde441f05efae67cfaec34661205711ac7031e1"}
Oct 30 00:23:36.669620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.669254    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl" event={"ID":"402d330d-6644-40e9-9bbe-8775bd0fadc4","Type":"ContainerDied","Data":"cba71bfdc100ccbe2d8ad362b04b61da46bdc4d94482afaa91567fb6788fd6dd"}
Oct 30 00:23:36.669620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.669283    4441 scope.go:122] "RemoveContainer" containerID="43e63ed303a196c7b5fe2985cdde441f05efae67cfaec34661205711ac7031e1"
Oct 30 00:23:36.679694 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.679678    4441 scope.go:122] "RemoveContainer" containerID="43e63ed303a196c7b5fe2985cdde441f05efae67cfaec34661205711ac7031e1"
Oct 30 00:23:36.679922 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:23:36.679904    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e63ed303a196c7b5fe2985cdde441f05efae67cfaec34661205711ac7031e1\": container with ID starting with 43e63ed303a196c7b5fe2985cdde441f05efae67cfaec34661205711ac7031e1 not found: ID does not exist" containerID="43e63ed303a196c7b5fe2985cdde441f05efae67cfaec34661205711ac7031e1"
Oct 30 00:23:36.679961 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.679934    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e63ed303a196c7b5fe2985cdde441f05efae67cfaec34661205711ac7031e1"} err="failed to get container status \"43e63ed303a196c7b5fe2985cdde441f05efae67cfaec34661205711ac7031e1\": rpc error: code = NotFound desc = could not find container \"43e63ed303a196c7b5fe2985cdde441f05efae67cfaec34661205711ac7031e1\": container with ID starting with 43e63ed303a196c7b5fe2985cdde441f05efae67cfaec34661205711ac7031e1 not found: ID does not exist"
Oct 30 00:23:36.720793 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.720770    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"]
Oct 30 00:23:36.726599 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:36.726579    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-x6wxl"]
Oct 30 00:23:38.610874 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:38.610845    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="402d330d-6644-40e9-9bbe-8775bd0fadc4" path="/var/lib/kubelet/pods/402d330d-6644-40e9-9bbe-8775bd0fadc4/volumes"
Oct 30 00:23:48.860614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.860570    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"]
Oct 30 00:23:48.861560 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.861536    4441 cpu_manager.go:414] "containerMap: removing container" podUID="402d330d-6644-40e9-9bbe-8775bd0fadc4" containerName="maas-api"
Oct 30 00:23:48.861560 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.861559    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402d330d-6644-40e9-9bbe-8775bd0fadc4" containerName="maas-api"
Oct 30 00:23:48.861766 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.861752    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="402d330d-6644-40e9-9bbe-8775bd0fadc4" containerName="maas-api"
Oct 30 00:23:48.864735 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.864715    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"
Oct 30 00:23:48.878710 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.878691    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0847e8c0\""
Oct 30 00:23:48.892437 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.892413    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xz6k\" (UniqueName: \"kubernetes.io/projected/65f4a9c0-bdd6-45d7-b363-f64e7461530c-kube-api-access-7xz6k\") pod \"payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq\" (UID: \"65f4a9c0-bdd6-45d7-b363-f64e7461530c\") " pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"
Oct 30 00:23:48.892620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.892605    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f4a9c0-bdd6-45d7-b363-f64e7461530c-plugins-config-volume\") pod \"payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq\" (UID: \"65f4a9c0-bdd6-45d7-b363-f64e7461530c\") " pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"
Oct 30 00:23:48.929970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.929945    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"]
Oct 30 00:23:48.934204 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.934185    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"
Oct 30 00:23:48.948181 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.948163    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0847e8c0-dockercfg-q4vkg\""
Oct 30 00:23:48.994646 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.994599    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xz6k\" (UniqueName: \"kubernetes.io/projected/65f4a9c0-bdd6-45d7-b363-f64e7461530c-kube-api-access-7xz6k\") pod \"payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq\" (UID: \"65f4a9c0-bdd6-45d7-b363-f64e7461530c\") " pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"
Oct 30 00:23:48.994863 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.994740    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350647e3-09d0-481a-baaf-dc0043c75ba7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n\" (UID: \"350647e3-09d0-481a-baaf-dc0043c75ba7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"
Oct 30 00:23:48.994973 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.994903    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f4a9c0-bdd6-45d7-b363-f64e7461530c-plugins-config-volume\") pod \"payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq\" (UID: \"65f4a9c0-bdd6-45d7-b363-f64e7461530c\") " pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"
Oct 30 00:23:48.995128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.995085    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lngb\" (UniqueName: \"kubernetes.io/projected/350647e3-09d0-481a-baaf-dc0043c75ba7-kube-api-access-4lngb\") pod \"payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n\" (UID: \"350647e3-09d0-481a-baaf-dc0043c75ba7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"
Oct 30 00:23:48.995900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:48.995880    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f4a9c0-bdd6-45d7-b363-f64e7461530c-plugins-config-volume\") pod \"payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq\" (UID: \"65f4a9c0-bdd6-45d7-b363-f64e7461530c\") " pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"
Oct 30 00:23:49.043045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.042771    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xz6k\" (UniqueName: \"kubernetes.io/projected/65f4a9c0-bdd6-45d7-b363-f64e7461530c-kube-api-access-7xz6k\") pod \"payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq\" (UID: \"65f4a9c0-bdd6-45d7-b363-f64e7461530c\") " pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"
Oct 30 00:23:49.100049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.099827    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lngb\" (UniqueName: \"kubernetes.io/projected/350647e3-09d0-481a-baaf-dc0043c75ba7-kube-api-access-4lngb\") pod \"payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n\" (UID: \"350647e3-09d0-481a-baaf-dc0043c75ba7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"
Oct 30 00:23:49.100254 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.100108    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350647e3-09d0-481a-baaf-dc0043c75ba7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n\" (UID: \"350647e3-09d0-481a-baaf-dc0043c75ba7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"
Oct 30 00:23:49.119110 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.101258    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350647e3-09d0-481a-baaf-dc0043c75ba7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n\" (UID: \"350647e3-09d0-481a-baaf-dc0043c75ba7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"
Oct 30 00:23:49.145855 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.145821    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lngb\" (UniqueName: \"kubernetes.io/projected/350647e3-09d0-481a-baaf-dc0043c75ba7-kube-api-access-4lngb\") pod \"payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n\" (UID: \"350647e3-09d0-481a-baaf-dc0043c75ba7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"
Oct 30 00:23:49.186045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.180353    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"
Oct 30 00:23:49.249612 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.249287    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"
Oct 30 00:23:49.314248 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.314216    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"]
Oct 30 00:23:49.314446 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.314420    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l" podUID="0c3f287b-5853-45dc-a051-3238bdcdf0ed" containerName="maas-api" containerID="cri-o://26c4219d664546e7f975c58f0b80489ae0f0698d5554cbcaf7effddd1e2fc85d" gracePeriod=30
Oct 30 00:23:49.656577 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.656550    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:23:49.715659 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.715570    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c3f287b-5853-45dc-a051-3238bdcdf0ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3f287b-5853-45dc-a051-3238bdcdf0ed-openshift-service-ca\") pod \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") "
Oct 30 00:23:49.717198 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.717167    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-maas-api-tls\") pod \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") "
Oct 30 00:23:49.717509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.717490    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3f287b-5853-45dc-a051-3238bdcdf0ed-kube-api-access-p475n\" (UniqueName: \"kubernetes.io/projected/0c3f287b-5853-45dc-a051-3238bdcdf0ed-kube-api-access-p475n\") pod \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") "
Oct 30 00:23:49.717692 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.717677    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-metrics-certs\") pod \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\" (UID: \"0c3f287b-5853-45dc-a051-3238bdcdf0ed\") "
Oct 30 00:23:49.736118 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.733540    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3f287b-5853-45dc-a051-3238bdcdf0ed-openshift-service-ca" pod "0c3f287b-5853-45dc-a051-3238bdcdf0ed" (UID: "0c3f287b-5853-45dc-a051-3238bdcdf0ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:23:49.736356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.736276    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-maas-api-tls" pod "0c3f287b-5853-45dc-a051-3238bdcdf0ed" (UID: "0c3f287b-5853-45dc-a051-3238bdcdf0ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:23:49.736356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.736319    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3f287b-5853-45dc-a051-3238bdcdf0ed-kube-api-access-p475n" pod "0c3f287b-5853-45dc-a051-3238bdcdf0ed" (UID: "0c3f287b-5853-45dc-a051-3238bdcdf0ed"). InnerVolumeSpecName "kube-api-access-p475n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:23:49.743450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.737873    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-metrics-certs" pod "0c3f287b-5853-45dc-a051-3238bdcdf0ed" (UID: "0c3f287b-5853-45dc-a051-3238bdcdf0ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:23:49.788532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.788498    4441 generic.go:356] "Generic (PLEG): container finished" podID="0c3f287b-5853-45dc-a051-3238bdcdf0ed" containerID="26c4219d664546e7f975c58f0b80489ae0f0698d5554cbcaf7effddd1e2fc85d" exitCode=0
Oct 30 00:23:49.788721 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.788549    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"
Oct 30 00:23:49.788721 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.788581    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l" event={"ID":"0c3f287b-5853-45dc-a051-3238bdcdf0ed","Type":"ContainerDied","Data":"26c4219d664546e7f975c58f0b80489ae0f0698d5554cbcaf7effddd1e2fc85d"}
Oct 30 00:23:49.788721 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.788640    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l" event={"ID":"0c3f287b-5853-45dc-a051-3238bdcdf0ed","Type":"ContainerDied","Data":"156249bc988ecf00f1d9df821a1a7f84902c165282ab5cd8f5f9441a01569d20"}
Oct 30 00:23:49.788721 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.788663    4441 scope.go:122] "RemoveContainer" containerID="26c4219d664546e7f975c58f0b80489ae0f0698d5554cbcaf7effddd1e2fc85d"
Oct 30 00:23:49.800243 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.800226    4441 scope.go:122] "RemoveContainer" containerID="26c4219d664546e7f975c58f0b80489ae0f0698d5554cbcaf7effddd1e2fc85d"
Oct 30 00:23:49.800530 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:23:49.800511    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c4219d664546e7f975c58f0b80489ae0f0698d5554cbcaf7effddd1e2fc85d\": container with ID starting with 26c4219d664546e7f975c58f0b80489ae0f0698d5554cbcaf7effddd1e2fc85d not found: ID does not exist" containerID="26c4219d664546e7f975c58f0b80489ae0f0698d5554cbcaf7effddd1e2fc85d"
Oct 30 00:23:49.800583 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.800541    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c4219d664546e7f975c58f0b80489ae0f0698d5554cbcaf7effddd1e2fc85d"} err="failed to get container status \"26c4219d664546e7f975c58f0b80489ae0f0698d5554cbcaf7effddd1e2fc85d\": rpc error: code = NotFound desc = could not find container \"26c4219d664546e7f975c58f0b80489ae0f0698d5554cbcaf7effddd1e2fc85d\": container with ID starting with 26c4219d664546e7f975c58f0b80489ae0f0698d5554cbcaf7effddd1e2fc85d not found: ID does not exist"
Oct 30 00:23:49.819883 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.819856    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p475n\" (UniqueName: \"kubernetes.io/projected/0c3f287b-5853-45dc-a051-3238bdcdf0ed-kube-api-access-p475n\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:49.819883 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.819882    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:49.820062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.819895    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3f287b-5853-45dc-a051-3238bdcdf0ed-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:49.820062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.819904    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3f287b-5853-45dc-a051-3238bdcdf0ed-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:23:49.842688 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.842663    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"]
Oct 30 00:23:49.847860 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:49.847839    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7417dc-dd4f776f5-xpf5l"]
Oct 30 00:23:50.622094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:50.616042    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c3f287b-5853-45dc-a051-3238bdcdf0ed" path="/var/lib/kubelet/pods/0c3f287b-5853-45dc-a051-3238bdcdf0ed/volumes"
Oct 30 00:23:53.040135 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.040101    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"]
Oct 30 00:23:53.041100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.041076    4441 cpu_manager.go:414] "containerMap: removing container" podUID="0c3f287b-5853-45dc-a051-3238bdcdf0ed" containerName="maas-api"
Oct 30 00:23:53.041230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.041099    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c3f287b-5853-45dc-a051-3238bdcdf0ed" containerName="maas-api"
Oct 30 00:23:53.041312 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.041298    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c3f287b-5853-45dc-a051-3238bdcdf0ed" containerName="maas-api"
Oct 30 00:23:53.045530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.045510    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:53.061575 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.061551    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6a55e1-serving-cert\""
Oct 30 00:23:53.184646 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.184614    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:53.184803 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.184784    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:53.184900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.184887    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zjfv\" (UniqueName: \"kubernetes.io/projected/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-kube-api-access-7zjfv\") pod \"maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:53.185016 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.184998    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-metrics-certs\") pod \"maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:53.286744 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.286714    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:53.286876 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.286795    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zjfv\" (UniqueName: \"kubernetes.io/projected/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-kube-api-access-7zjfv\") pod \"maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:53.286936 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.286875    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-metrics-certs\") pod \"maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:53.286980 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.286942    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:53.287533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.287511    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:53.289260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.289234    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-metrics-certs\") pod \"maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:53.289260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.289254    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:53.347095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.346749    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zjfv\" (UniqueName: \"kubernetes.io/projected/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-kube-api-access-7zjfv\") pod \"maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:53.372844 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:53.372808    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:23:59.592348 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:59.592298    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"]
Oct 30 00:23:59.592849 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:59.592548    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr" podUID="e2fd350b-e02c-488b-af30-a192c3d9c53f" containerName="maas-api" containerID="cri-o://97fdc76a71dc2301b57b2014c1b9bc64c86ef6fd132b8217cdeaecdad0e88d27" gracePeriod=30
Oct 30 00:23:59.947968 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:59.947933    4441 generic.go:356] "Generic (PLEG): container finished" podID="e2fd350b-e02c-488b-af30-a192c3d9c53f" containerID="97fdc76a71dc2301b57b2014c1b9bc64c86ef6fd132b8217cdeaecdad0e88d27" exitCode=0
Oct 30 00:23:59.948232 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:59.948206    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr" event={"ID":"e2fd350b-e02c-488b-af30-a192c3d9c53f","Type":"ContainerDied","Data":"97fdc76a71dc2301b57b2014c1b9bc64c86ef6fd132b8217cdeaecdad0e88d27"}
Oct 30 00:23:59.980870 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:23:59.980516    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:24:00.167198 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.167154    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2fd350b-e02c-488b-af30-a192c3d9c53f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2fd350b-e02c-488b-af30-a192c3d9c53f-openshift-service-ca\") pod \"e2fd350b-e02c-488b-af30-a192c3d9c53f\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") "
Oct 30 00:24:00.167394 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.167313    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-metrics-certs\") pod \"e2fd350b-e02c-488b-af30-a192c3d9c53f\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") "
Oct 30 00:24:00.167475 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.167405    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2fd350b-e02c-488b-af30-a192c3d9c53f-kube-api-access-vkczg\" (UniqueName: \"kubernetes.io/projected/e2fd350b-e02c-488b-af30-a192c3d9c53f-kube-api-access-vkczg\") pod \"e2fd350b-e02c-488b-af30-a192c3d9c53f\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") "
Oct 30 00:24:00.167536 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.167506    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-maas-api-tls\") pod \"e2fd350b-e02c-488b-af30-a192c3d9c53f\" (UID: \"e2fd350b-e02c-488b-af30-a192c3d9c53f\") "
Oct 30 00:24:00.167593 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.167566    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2fd350b-e02c-488b-af30-a192c3d9c53f-openshift-service-ca" pod "e2fd350b-e02c-488b-af30-a192c3d9c53f" (UID: "e2fd350b-e02c-488b-af30-a192c3d9c53f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:24:00.168590 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.168535    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2fd350b-e02c-488b-af30-a192c3d9c53f-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:00.169737 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.169705    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-metrics-certs" pod "e2fd350b-e02c-488b-af30-a192c3d9c53f" (UID: "e2fd350b-e02c-488b-af30-a192c3d9c53f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:24:00.169860 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.169824    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fd350b-e02c-488b-af30-a192c3d9c53f-kube-api-access-vkczg" pod "e2fd350b-e02c-488b-af30-a192c3d9c53f" (UID: "e2fd350b-e02c-488b-af30-a192c3d9c53f"). InnerVolumeSpecName "kube-api-access-vkczg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:24:00.169991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.169969    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-maas-api-tls" pod "e2fd350b-e02c-488b-af30-a192c3d9c53f" (UID: "e2fd350b-e02c-488b-af30-a192c3d9c53f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:24:00.270580 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.270540    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkczg\" (UniqueName: \"kubernetes.io/projected/e2fd350b-e02c-488b-af30-a192c3d9c53f-kube-api-access-vkczg\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:00.270838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.270820    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:00.270938 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.270926    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2fd350b-e02c-488b-af30-a192c3d9c53f-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:00.294453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.294419    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"]
Oct 30 00:24:00.296479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.296447    4441 cpu_manager.go:414] "containerMap: removing container" podUID="e2fd350b-e02c-488b-af30-a192c3d9c53f" containerName="maas-api"
Oct 30 00:24:00.296479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.296475    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2fd350b-e02c-488b-af30-a192c3d9c53f" containerName="maas-api"
Oct 30 00:24:00.296805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.296790    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2fd350b-e02c-488b-af30-a192c3d9c53f" containerName="maas-api"
Oct 30 00:24:00.300810 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.300788    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:00.314971 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.314937    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0847e8c0-serving-cert\""
Oct 30 00:24:00.487048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.483619    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-maas-api-tls\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-lrz57\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:00.487048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.484098    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xggl\" (UniqueName: \"kubernetes.io/projected/f9131a7c-3280-43e8-a388-e9c51a7a0389-kube-api-access-7xggl\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-lrz57\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:00.487048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.484208    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-metrics-certs\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-lrz57\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:00.487048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.484393    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9131a7c-3280-43e8-a388-e9c51a7a0389-openshift-service-ca\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-lrz57\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:00.567589 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.565628    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"]
Oct 30 00:24:00.596046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.586316    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-metrics-certs\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-lrz57\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:00.596046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.586442    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9131a7c-3280-43e8-a388-e9c51a7a0389-openshift-service-ca\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-lrz57\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:00.596046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.586636    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-maas-api-tls\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-lrz57\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:00.596046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.586793    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xggl\" (UniqueName: \"kubernetes.io/projected/f9131a7c-3280-43e8-a388-e9c51a7a0389-kube-api-access-7xggl\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-lrz57\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:00.596046 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:24:00.588136    4441 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-serving-cert: secret "maas-api-e2e-mt-0847e8c0-serving-cert" not found
Oct 30 00:24:00.596046 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:24:00.588203    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-maas-api-tls podName:f9131a7c-3280-43e8-a388-e9c51a7a0389 nodeName:}" failed. No retries permitted until 2026-10-30 00:24:01.088183664 +0000 UTC m=+1150.613477550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-maas-api-tls") pod "maas-api-e2e-mt-0847e8c0-585888c668-lrz57" (UID: "f9131a7c-3280-43e8-a388-e9c51a7a0389") : secret "maas-api-e2e-mt-0847e8c0-serving-cert" not found
Oct 30 00:24:00.596046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.588774    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9131a7c-3280-43e8-a388-e9c51a7a0389-openshift-service-ca\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-lrz57\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:00.596847 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.596638    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"]
Oct 30 00:24:00.607289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.607262    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"]
Oct 30 00:24:00.672247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.670981    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"]
Oct 30 00:24:00.672247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.671066    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"]
Oct 30 00:24:00.672247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.671098    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"]
Oct 30 00:24:00.672247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.671124    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"]
Oct 30 00:24:00.679997 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.679928    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-metrics-certs\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-lrz57\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:00.719147 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.702823    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xggl\" (UniqueName: \"kubernetes.io/projected/f9131a7c-3280-43e8-a388-e9c51a7a0389-kube-api-access-7xggl\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-lrz57\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:00.967632 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.966925    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr" event={"ID":"e2fd350b-e02c-488b-af30-a192c3d9c53f","Type":"ContainerDied","Data":"51a2e05f829bd3932b2b999a6a5a838aec47a2b419d4e44811f27a461bdbbf27"}
Oct 30 00:24:00.967632 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.967008    4441 scope.go:122] "RemoveContainer" containerID="97fdc76a71dc2301b57b2014c1b9bc64c86ef6fd132b8217cdeaecdad0e88d27"
Oct 30 00:24:00.967632 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:00.967208    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"
Oct 30 00:24:01.028236 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.028169    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"]
Oct 30 00:24:01.037850 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.037817    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e2a011-74f7d6fc9d-9vrsr"]
Oct 30 00:24:01.125353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.125263    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-maas-api-tls\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-lrz57\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:01.131891 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.131860    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-maas-api-tls\") pod \"maas-api-e2e-mt-0847e8c0-585888c668-lrz57\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:01.162619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.162585    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"]
Oct 30 00:24:01.217585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.217557    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:01.272889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.272245    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"]
Oct 30 00:24:01.377893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.376084    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"]
Oct 30 00:24:01.440545 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.440515    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"]
Oct 30 00:24:01.451802 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.451608    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"]
Oct 30 00:24:01.452848 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.452806    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"]
Oct 30 00:24:01.921860 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.921759    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"]
Oct 30 00:24:01.929630 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.928278    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"]
Oct 30 00:24:01.929630 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:01.928562    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr" podUID="32724ded-d541-4e35-bd55-58b781767acb" containerName="maas-api" containerID="cri-o://c66a0c60a153249d3d93c831c3fab338a583c8189c63e757ac82eb80a1c3694c" gracePeriod=30
Oct 30 00:24:02.060084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.059418    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57" event={"ID":"f9131a7c-3280-43e8-a388-e9c51a7a0389","Type":"ContainerStarted","Data":"611ee5feca68607ecfbe28f280a9ef6e0de06a21c2fb86f62a2dc3187ee087f8"}
Oct 30 00:24:02.073731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.073590    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n" event={"ID":"350647e3-09d0-481a-baaf-dc0043c75ba7","Type":"ContainerStarted","Data":"58ec3805eaee627c2e9cb1ef2d70a825b303a16ff22d572c7037629c9cd4c4e8"}
Oct 30 00:24:02.073731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.073642    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n" event={"ID":"350647e3-09d0-481a-baaf-dc0043c75ba7","Type":"ContainerStarted","Data":"a59cddd9cb26261218bb141c7461dfdc1cf3afd0bafaec93b9cb559d2316c6b8"}
Oct 30 00:24:02.073731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.073694    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"
Oct 30 00:24:02.078293 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.077289    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv" event={"ID":"96327c70-a528-47d6-ac5d-d68072291d6d","Type":"ContainerStarted","Data":"d819ad3220b50f468b6c9b3c1733d5a52427047a435d7bf2985759791a32f446"}
Oct 30 00:24:02.078293 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.077342    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv" event={"ID":"96327c70-a528-47d6-ac5d-d68072291d6d","Type":"ContainerStarted","Data":"1ed5c716e1d781550d8dd77dac873c619ea326bd5810fb3c0b83162ffe5d78e3"}
Oct 30 00:24:02.078293 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.078237    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"
Oct 30 00:24:02.080380 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.080281    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8" event={"ID":"02834ba0-2a7a-4e99-8d35-d9b15c9cef67","Type":"ContainerStarted","Data":"c296ce844115849d48c163f7d68246af8ac8f2e578d5cdd4ab473ef6ab36f2e9"}
Oct 30 00:24:02.083237 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.083193    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6" event={"ID":"ea8774a1-1532-46be-8af1-086a1d709c01","Type":"ContainerStarted","Data":"3b507d29b770a9189698ad025a3b2e514bd2ed0ddb90199bf0efaee1e04a4bc6"}
Oct 30 00:24:02.084124 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.084080    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"
Oct 30 00:24:02.086280 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.086253    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7" event={"ID":"caaf8af0-c2b6-4544-896e-fc9cf76e0e99","Type":"ContainerStarted","Data":"c6ae40b0fbd554762660bedd54ad16416b477442832ea1d72c1ec03d540f18fe"}
Oct 30 00:24:02.090139 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.090117    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq" event={"ID":"65f4a9c0-bdd6-45d7-b363-f64e7461530c","Type":"ContainerStarted","Data":"ed4d14a53f9ce6d795cfff0e11a894955a1af0199217b7cb4e07664215cf9bf8"}
Oct 30 00:24:02.090889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.090871    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"
Oct 30 00:24:02.138755 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.138694    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n" podStartSLOduration=14.13867687 podStartE2EDuration="14.13867687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:02.134565953 +0000 UTC m=+1151.659859830" watchObservedRunningTime="2026-10-30 00:24:02.13867687 +0000 UTC m=+1151.663970749"
Oct 30 00:24:02.180759 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.180692    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv" podStartSLOduration=29.180671914 podStartE2EDuration="29.180671914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:02.176400931 +0000 UTC m=+1151.701694808" watchObservedRunningTime="2026-10-30 00:24:02.180671914 +0000 UTC m=+1151.705965792"
Oct 30 00:24:02.221675 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.220441    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6" podStartSLOduration=27.220423287 podStartE2EDuration="27.220423287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:02.215144341 +0000 UTC m=+1151.740438217" watchObservedRunningTime="2026-10-30 00:24:02.220423287 +0000 UTC m=+1151.745717233"
Oct 30 00:24:02.256264 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.256203    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq" podStartSLOduration=14.256184762 podStartE2EDuration="14.256184762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:02.253533454 +0000 UTC m=+1151.778827333" watchObservedRunningTime="2026-10-30 00:24:02.256184762 +0000 UTC m=+1151.781478640"
Oct 30 00:24:02.358218 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.357851    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:24:02.469867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.469282    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32724ded-d541-4e35-bd55-58b781767acb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32724ded-d541-4e35-bd55-58b781767acb-openshift-service-ca\") pod \"32724ded-d541-4e35-bd55-58b781767acb\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") "
Oct 30 00:24:02.471520 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.471421    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-metrics-certs\") pod \"32724ded-d541-4e35-bd55-58b781767acb\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") "
Oct 30 00:24:02.471952 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.471890    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32724ded-d541-4e35-bd55-58b781767acb-kube-api-access-29vmk\" (UniqueName: \"kubernetes.io/projected/32724ded-d541-4e35-bd55-58b781767acb-kube-api-access-29vmk\") pod \"32724ded-d541-4e35-bd55-58b781767acb\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") "
Oct 30 00:24:02.474104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.474081    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-maas-api-tls\") pod \"32724ded-d541-4e35-bd55-58b781767acb\" (UID: \"32724ded-d541-4e35-bd55-58b781767acb\") "
Oct 30 00:24:02.475826 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.472663    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32724ded-d541-4e35-bd55-58b781767acb-openshift-service-ca" pod "32724ded-d541-4e35-bd55-58b781767acb" (UID: "32724ded-d541-4e35-bd55-58b781767acb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:24:02.476480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.476266    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32724ded-d541-4e35-bd55-58b781767acb-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:02.481391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.481368    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-metrics-certs" pod "32724ded-d541-4e35-bd55-58b781767acb" (UID: "32724ded-d541-4e35-bd55-58b781767acb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:24:02.486992 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.483087    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32724ded-d541-4e35-bd55-58b781767acb-kube-api-access-29vmk" pod "32724ded-d541-4e35-bd55-58b781767acb" (UID: "32724ded-d541-4e35-bd55-58b781767acb"). InnerVolumeSpecName "kube-api-access-29vmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:24:02.487335 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.487195    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-maas-api-tls" pod "32724ded-d541-4e35-bd55-58b781767acb" (UID: "32724ded-d541-4e35-bd55-58b781767acb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:24:02.578725 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.578691    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:02.578725 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.578721    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29vmk\" (UniqueName: \"kubernetes.io/projected/32724ded-d541-4e35-bd55-58b781767acb-kube-api-access-29vmk\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:02.578725 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.578731    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32724ded-d541-4e35-bd55-58b781767acb-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:02.614765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:02.614736    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2fd350b-e02c-488b-af30-a192c3d9c53f" path="/var/lib/kubelet/pods/e2fd350b-e02c-488b-af30-a192c3d9c53f/volumes"
Oct 30 00:24:03.101375 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.101328    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8" event={"ID":"02834ba0-2a7a-4e99-8d35-d9b15c9cef67","Type":"ContainerStarted","Data":"f570617066c57f7b011066c97c15385c3de77da3c7e8cb1d6d9d560e802e37fe"}
Oct 30 00:24:03.101865 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.101579    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:24:03.103567 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.103513    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6" event={"ID":"ea8774a1-1532-46be-8af1-086a1d709c01","Type":"ContainerStarted","Data":"0fbe58969ada3e96ae961eadcd582b3a5136cbfd702980e2ea8994015a43cf3f"}
Oct 30 00:24:03.105356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.105329    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7" event={"ID":"caaf8af0-c2b6-4544-896e-fc9cf76e0e99","Type":"ContainerStarted","Data":"f068bab5dc895674f8439f2846a82aa00c61f9e5e831c04dc972a2732036a03e"}
Oct 30 00:24:03.105478 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.105430    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"
Oct 30 00:24:03.106802 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.106771    4441 generic.go:356] "Generic (PLEG): container finished" podID="32724ded-d541-4e35-bd55-58b781767acb" containerID="c66a0c60a153249d3d93c831c3fab338a583c8189c63e757ac82eb80a1c3694c" exitCode=0
Oct 30 00:24:03.106932 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.106812    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"
Oct 30 00:24:03.106932 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.106884    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr" event={"ID":"32724ded-d541-4e35-bd55-58b781767acb","Type":"ContainerDied","Data":"c66a0c60a153249d3d93c831c3fab338a583c8189c63e757ac82eb80a1c3694c"}
Oct 30 00:24:03.106932 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.106924    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr" event={"ID":"32724ded-d541-4e35-bd55-58b781767acb","Type":"ContainerDied","Data":"4597d3c82918e0329602b151dedaf1904f0f8b7e753198477d1c11df96d9a734"}
Oct 30 00:24:03.107112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.106947    4441 scope.go:122] "RemoveContainer" containerID="c66a0c60a153249d3d93c831c3fab338a583c8189c63e757ac82eb80a1c3694c"
Oct 30 00:24:03.108877 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.108856    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq" event={"ID":"65f4a9c0-bdd6-45d7-b363-f64e7461530c","Type":"ContainerStarted","Data":"e0fca206ead763d19ebf62f23d3cb25e9b317ea58a22061051114426e35f2c01"}
Oct 30 00:24:03.110891 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.110869    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57" event={"ID":"f9131a7c-3280-43e8-a388-e9c51a7a0389","Type":"ContainerStarted","Data":"785921c5788f789d12126ab325e03f2b22906183258a336dce129db25e5fcae1"}
Oct 30 00:24:03.111152 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.111135    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:03.118189 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.118173    4441 scope.go:122] "RemoveContainer" containerID="c66a0c60a153249d3d93c831c3fab338a583c8189c63e757ac82eb80a1c3694c"
Oct 30 00:24:03.118454 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:24:03.118431    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66a0c60a153249d3d93c831c3fab338a583c8189c63e757ac82eb80a1c3694c\": container with ID starting with c66a0c60a153249d3d93c831c3fab338a583c8189c63e757ac82eb80a1c3694c not found: ID does not exist" containerID="c66a0c60a153249d3d93c831c3fab338a583c8189c63e757ac82eb80a1c3694c"
Oct 30 00:24:03.118527 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.118466    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66a0c60a153249d3d93c831c3fab338a583c8189c63e757ac82eb80a1c3694c"} err="failed to get container status \"c66a0c60a153249d3d93c831c3fab338a583c8189c63e757ac82eb80a1c3694c\": rpc error: code = NotFound desc = could not find container \"c66a0c60a153249d3d93c831c3fab338a583c8189c63e757ac82eb80a1c3694c\": container with ID starting with c66a0c60a153249d3d93c831c3fab338a583c8189c63e757ac82eb80a1c3694c not found: ID does not exist"
Oct 30 00:24:03.145539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.145487    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8" podStartSLOduration=9.505018991 podStartE2EDuration="10.145469917s" totalImagesPullingTime="640.450926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:03.14280135 +0000 UTC m=+1152.668095230" watchObservedRunningTime="2026-10-30 00:24:03.145469917 +0000 UTC m=+1152.670763796"
Oct 30 00:24:03.186316 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.186254    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7" podStartSLOduration=30.186235162 podStartE2EDuration="30.186235162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:03.181764607 +0000 UTC m=+1152.707058487" watchObservedRunningTime="2026-10-30 00:24:03.186235162 +0000 UTC m=+1152.711529040"
Oct 30 00:24:03.228092 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.228015    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57" podStartSLOduration=2.8615406500000002 podStartE2EDuration="3.227998787s" totalImagesPullingTime="366.458137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:03.223159406 +0000 UTC m=+1152.748453284" watchObservedRunningTime="2026-10-30 00:24:03.227998787 +0000 UTC m=+1152.753292664"
Oct 30 00:24:03.261320 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.261286    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"]
Oct 30 00:24:03.265679 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:03.265649    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-461609-6b675c587b-58jkr"]
Oct 30 00:24:03.903686 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:24:03.903646    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:24:04.614293 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:04.614260    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32724ded-d541-4e35-bd55-58b781767acb" path="/var/lib/kubelet/pods/32724ded-d541-4e35-bd55-58b781767acb/volumes"
Oct 30 00:24:07.882799 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:07.882576    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"]
Oct 30 00:24:07.883477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:07.882950    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" podUID="71d7723e-9a7d-4858-8093-4b13593809db" containerName="istio-proxy" containerID="cri-o://0a2f21822a04a80ef3e54e47bf9bdab5f353b4317c81226ca89e8e203e5eb6b1" gracePeriod=30
Oct 30 00:24:09.152447 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:09.152402    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:24:09.161347 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:09.161315    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:24:13.121553 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.114816    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"
Oct 30 00:24:13.228611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.228583    4441 generic.go:356] "Generic (PLEG): container finished" podID="71d7723e-9a7d-4858-8093-4b13593809db" containerID="0a2f21822a04a80ef3e54e47bf9bdab5f353b4317c81226ca89e8e203e5eb6b1" exitCode=0
Oct 30 00:24:13.228791 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.228776    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" event={"ID":"71d7723e-9a7d-4858-8093-4b13593809db","Type":"ContainerDied","Data":"0a2f21822a04a80ef3e54e47bf9bdab5f353b4317c81226ca89e8e203e5eb6b1"}
Oct 30 00:24:13.228854 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.228817    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd" event={"ID":"71d7723e-9a7d-4858-8093-4b13593809db","Type":"ContainerDied","Data":"65779125364d53a73f0b4882727b9ad525f421b496f94ef63f11849b6ba8a6b0"}
Oct 30 00:24:13.228854 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.228834    4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65779125364d53a73f0b4882727b9ad525f421b496f94ef63f11849b6ba8a6b0"
Oct 30 00:24:13.228854 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.228847    4441 scope.go:122] "RemoveContainer" containerID="e556283aabfe42c525e73a2e1fde7c897cbaa541a5c71f0adcdb2197190e53e9"
Oct 30 00:24:13.242320 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.242293    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:24:13.283533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.283502    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-data\") pod \"71d7723e-9a7d-4858-8093-4b13593809db\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") "
Oct 30 00:24:13.283739 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.283566    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-istio-token\" (UniqueName: \"kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-istio-token\") pod \"71d7723e-9a7d-4858-8093-4b13593809db\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") "
Oct 30 00:24:13.283739 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.283731    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d7723e-9a7d-4858-8093-4b13593809db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d7723e-9a7d-4858-8093-4b13593809db-istiod-ca-cert\") pod \"71d7723e-9a7d-4858-8093-4b13593809db\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") "
Oct 30 00:24:13.283872 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.283787    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-socket\") pod \"71d7723e-9a7d-4858-8093-4b13593809db\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") "
Oct 30 00:24:13.283872 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.283836    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-data" pod "71d7723e-9a7d-4858-8093-4b13593809db" (UID: "71d7723e-9a7d-4858-8093-4b13593809db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:24:13.284215 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.283891    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-kube-api-access-7xqwj\" (UniqueName: \"kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-kube-api-access-7xqwj\") pod \"71d7723e-9a7d-4858-8093-4b13593809db\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") "
Oct 30 00:24:13.284215 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.284128    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-credential-socket\") pod \"71d7723e-9a7d-4858-8093-4b13593809db\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") "
Oct 30 00:24:13.284444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.284219    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-envoy\") pod \"71d7723e-9a7d-4858-8093-4b13593809db\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") "
Oct 30 00:24:13.284444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.284277    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-certs\") pod \"71d7723e-9a7d-4858-8093-4b13593809db\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") "
Oct 30 00:24:13.284444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.284308    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71d7723e-9a7d-4858-8093-4b13593809db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d7723e-9a7d-4858-8093-4b13593809db-istio-podinfo\") pod \"71d7723e-9a7d-4858-8093-4b13593809db\" (UID: \"71d7723e-9a7d-4858-8093-4b13593809db\") "
Oct 30 00:24:13.284605 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.284125    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-socket" pod "71d7723e-9a7d-4858-8093-4b13593809db" (UID: "71d7723e-9a7d-4858-8093-4b13593809db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:24:13.284605 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.284310    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-credential-socket" pod "71d7723e-9a7d-4858-8093-4b13593809db" (UID: "71d7723e-9a7d-4858-8093-4b13593809db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:24:13.284819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.284791    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-certs" pod "71d7723e-9a7d-4858-8093-4b13593809db" (UID: "71d7723e-9a7d-4858-8093-4b13593809db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:24:13.285671 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.285275    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:13.285671 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.285298    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:13.285671 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.285313    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:13.285671 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.285326    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:13.287934 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.287904    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d7723e-9a7d-4858-8093-4b13593809db-istiod-ca-cert" pod "71d7723e-9a7d-4858-8093-4b13593809db" (UID: "71d7723e-9a7d-4858-8093-4b13593809db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:24:13.288547 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.288524    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-istio-token" pod "71d7723e-9a7d-4858-8093-4b13593809db" (UID: "71d7723e-9a7d-4858-8093-4b13593809db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:24:13.288547 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.288530    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-envoy" pod "71d7723e-9a7d-4858-8093-4b13593809db" (UID: "71d7723e-9a7d-4858-8093-4b13593809db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:24:13.289432 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.289394    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71d7723e-9a7d-4858-8093-4b13593809db-istio-podinfo" pod "71d7723e-9a7d-4858-8093-4b13593809db" (UID: "71d7723e-9a7d-4858-8093-4b13593809db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:24:13.292228 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.292196    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-kube-api-access-7xqwj" pod "71d7723e-9a7d-4858-8093-4b13593809db" (UID: "71d7723e-9a7d-4858-8093-4b13593809db"). InnerVolumeSpecName "kube-api-access-7xqwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:24:13.386862 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.386819    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:13.386862 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.386854    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d7723e-9a7d-4858-8093-4b13593809db-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:13.387108 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.386869    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xqwj\" (UniqueName: \"kubernetes.io/projected/71d7723e-9a7d-4858-8093-4b13593809db-kube-api-access-7xqwj\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:13.387108 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.386884    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d7723e-9a7d-4858-8093-4b13593809db-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:13.387108 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:13.386895    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d7723e-9a7d-4858-8093-4b13593809db-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:14.132020 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:14.131991    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"
Oct 30 00:24:14.140633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:14.140608    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"
Oct 30 00:24:14.161826 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:14.161803    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"
Oct 30 00:24:14.171327 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:14.171304    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"
Oct 30 00:24:14.258641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:14.258607    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"
Oct 30 00:24:14.347118 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:14.347079    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"]
Oct 30 00:24:14.354225 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:14.354197    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e2a011-openshift-default-787f9c9557-dnnrd"]
Oct 30 00:24:14.610628 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:14.610596    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d7723e-9a7d-4858-8093-4b13593809db" path="/var/lib/kubelet/pods/71d7723e-9a7d-4858-8093-4b13593809db/volumes"
Oct 30 00:24:18.961074 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:18.961017    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:24:22.783575 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:22.783545    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"]
Oct 30 00:24:22.821593 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:22.821559    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"]
Oct 30 00:24:23.877077 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:23.877043    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"]
Oct 30 00:24:27.784845 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:27.784784    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7" podUID="caaf8af0-c2b6-4544-896e-fc9cf76e0e99" containerName="payload-processing" containerID="cri-o://f068bab5dc895674f8439f2846a82aa00c61f9e5e831c04dc972a2732036a03e" gracePeriod=25
Oct 30 00:24:27.822772 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:27.822708    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv" podUID="96327c70-a528-47d6-ac5d-d68072291d6d" containerName="payload-pre-processing" containerID="cri-o://d819ad3220b50f468b6c9b3c1733d5a52427047a435d7bf2985759791a32f446" gracePeriod=25
Oct 30 00:24:28.149282 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.149252    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"
Oct 30 00:24:28.203497 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.203465    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-plugins-config-volume\") pod \"caaf8af0-c2b6-4544-896e-fc9cf76e0e99\" (UID: \"caaf8af0-c2b6-4544-896e-fc9cf76e0e99\") "
Oct 30 00:24:28.203656 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.203550    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-kube-api-access-4t9v7\" (UniqueName: \"kubernetes.io/projected/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-kube-api-access-4t9v7\") pod \"caaf8af0-c2b6-4544-896e-fc9cf76e0e99\" (UID: \"caaf8af0-c2b6-4544-896e-fc9cf76e0e99\") "
Oct 30 00:24:28.203910 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.203886    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-plugins-config-volume" pod "caaf8af0-c2b6-4544-896e-fc9cf76e0e99" (UID: "caaf8af0-c2b6-4544-896e-fc9cf76e0e99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:24:28.204597 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.204568    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:28.205871 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.205848    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-kube-api-access-4t9v7" pod "caaf8af0-c2b6-4544-896e-fc9cf76e0e99" (UID: "caaf8af0-c2b6-4544-896e-fc9cf76e0e99"). InnerVolumeSpecName "kube-api-access-4t9v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:24:28.207818 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.207798    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"
Oct 30 00:24:28.305516 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.305478    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96327c70-a528-47d6-ac5d-d68072291d6d-kube-api-access-wwtbs\" (UniqueName: \"kubernetes.io/projected/96327c70-a528-47d6-ac5d-d68072291d6d-kube-api-access-wwtbs\") pod \"96327c70-a528-47d6-ac5d-d68072291d6d\" (UID: \"96327c70-a528-47d6-ac5d-d68072291d6d\") "
Oct 30 00:24:28.305686 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.305609    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96327c70-a528-47d6-ac5d-d68072291d6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96327c70-a528-47d6-ac5d-d68072291d6d-plugins-config-volume\") pod \"96327c70-a528-47d6-ac5d-d68072291d6d\" (UID: \"96327c70-a528-47d6-ac5d-d68072291d6d\") "
Oct 30 00:24:28.306371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.306311    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96327c70-a528-47d6-ac5d-d68072291d6d-plugins-config-volume" pod "96327c70-a528-47d6-ac5d-d68072291d6d" (UID: "96327c70-a528-47d6-ac5d-d68072291d6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:24:28.306561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.306537    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96327c70-a528-47d6-ac5d-d68072291d6d-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:28.306659 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.306568    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9v7\" (UniqueName: \"kubernetes.io/projected/caaf8af0-c2b6-4544-896e-fc9cf76e0e99-kube-api-access-4t9v7\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:28.308677 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.308643    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96327c70-a528-47d6-ac5d-d68072291d6d-kube-api-access-wwtbs" pod "96327c70-a528-47d6-ac5d-d68072291d6d" (UID: "96327c70-a528-47d6-ac5d-d68072291d6d"). InnerVolumeSpecName "kube-api-access-wwtbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:24:28.408008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.407925    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwtbs\" (UniqueName: \"kubernetes.io/projected/96327c70-a528-47d6-ac5d-d68072291d6d-kube-api-access-wwtbs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:28.417981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.417943    4441 generic.go:356] "Generic (PLEG): container finished" podID="caaf8af0-c2b6-4544-896e-fc9cf76e0e99" containerID="f068bab5dc895674f8439f2846a82aa00c61f9e5e831c04dc972a2732036a03e" exitCode=0
Oct 30 00:24:28.418157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.417998    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"
Oct 30 00:24:28.418157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.417993    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7" event={"ID":"caaf8af0-c2b6-4544-896e-fc9cf76e0e99","Type":"ContainerDied","Data":"f068bab5dc895674f8439f2846a82aa00c61f9e5e831c04dc972a2732036a03e"}
Oct 30 00:24:28.418278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.418189    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7" event={"ID":"caaf8af0-c2b6-4544-896e-fc9cf76e0e99","Type":"ContainerDied","Data":"c6ae40b0fbd554762660bedd54ad16416b477442832ea1d72c1ec03d540f18fe"}
Oct 30 00:24:28.418278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.418219    4441 scope.go:122] "RemoveContainer" containerID="f068bab5dc895674f8439f2846a82aa00c61f9e5e831c04dc972a2732036a03e"
Oct 30 00:24:28.419815 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.419790    4441 generic.go:356] "Generic (PLEG): container finished" podID="96327c70-a528-47d6-ac5d-d68072291d6d" containerID="d819ad3220b50f468b6c9b3c1733d5a52427047a435d7bf2985759791a32f446" exitCode=0
Oct 30 00:24:28.419926 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.419829    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"
Oct 30 00:24:28.419926 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.419875    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv" event={"ID":"96327c70-a528-47d6-ac5d-d68072291d6d","Type":"ContainerDied","Data":"d819ad3220b50f468b6c9b3c1733d5a52427047a435d7bf2985759791a32f446"}
Oct 30 00:24:28.420086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.419928    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv" event={"ID":"96327c70-a528-47d6-ac5d-d68072291d6d","Type":"ContainerDied","Data":"1ed5c716e1d781550d8dd77dac873c619ea326bd5810fb3c0b83162ffe5d78e3"}
Oct 30 00:24:28.432304 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.432287    4441 scope.go:122] "RemoveContainer" containerID="f068bab5dc895674f8439f2846a82aa00c61f9e5e831c04dc972a2732036a03e"
Oct 30 00:24:28.432594 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:24:28.432576    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f068bab5dc895674f8439f2846a82aa00c61f9e5e831c04dc972a2732036a03e\": container with ID starting with f068bab5dc895674f8439f2846a82aa00c61f9e5e831c04dc972a2732036a03e not found: ID does not exist" containerID="f068bab5dc895674f8439f2846a82aa00c61f9e5e831c04dc972a2732036a03e"
Oct 30 00:24:28.432652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.432600    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f068bab5dc895674f8439f2846a82aa00c61f9e5e831c04dc972a2732036a03e"} err="failed to get container status \"f068bab5dc895674f8439f2846a82aa00c61f9e5e831c04dc972a2732036a03e\": rpc error: code = NotFound desc = could not find container \"f068bab5dc895674f8439f2846a82aa00c61f9e5e831c04dc972a2732036a03e\": container with ID starting with f068bab5dc895674f8439f2846a82aa00c61f9e5e831c04dc972a2732036a03e not found: ID does not exist"
Oct 30 00:24:28.432652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.432623    4441 scope.go:122] "RemoveContainer" containerID="d819ad3220b50f468b6c9b3c1733d5a52427047a435d7bf2985759791a32f446"
Oct 30 00:24:28.443596 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.443566    4441 scope.go:122] "RemoveContainer" containerID="d819ad3220b50f468b6c9b3c1733d5a52427047a435d7bf2985759791a32f446"
Oct 30 00:24:28.443896 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:24:28.443868    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d819ad3220b50f468b6c9b3c1733d5a52427047a435d7bf2985759791a32f446\": container with ID starting with d819ad3220b50f468b6c9b3c1733d5a52427047a435d7bf2985759791a32f446 not found: ID does not exist" containerID="d819ad3220b50f468b6c9b3c1733d5a52427047a435d7bf2985759791a32f446"
Oct 30 00:24:28.443954 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.443900    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d819ad3220b50f468b6c9b3c1733d5a52427047a435d7bf2985759791a32f446"} err="failed to get container status \"d819ad3220b50f468b6c9b3c1733d5a52427047a435d7bf2985759791a32f446\": rpc error: code = NotFound desc = could not find container \"d819ad3220b50f468b6c9b3c1733d5a52427047a435d7bf2985759791a32f446\": container with ID starting with d819ad3220b50f468b6c9b3c1733d5a52427047a435d7bf2985759791a32f446 not found: ID does not exist"
Oct 30 00:24:28.478309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.478287    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"]
Oct 30 00:24:28.482057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.482035    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-596798b9bd-k4dc7"]
Oct 30 00:24:28.509384 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.509356    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"]
Oct 30 00:24:28.514774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.514750    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-66cfdbc5f-m6tcv"]
Oct 30 00:24:28.611345 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.611315    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96327c70-a528-47d6-ac5d-d68072291d6d" path="/var/lib/kubelet/pods/96327c70-a528-47d6-ac5d-d68072291d6d/volumes"
Oct 30 00:24:28.611785 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.611770    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caaf8af0-c2b6-4544-896e-fc9cf76e0e99" path="/var/lib/kubelet/pods/caaf8af0-c2b6-4544-896e-fc9cf76e0e99/volumes"
Oct 30 00:24:28.878095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:28.878047    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6" podUID="ea8774a1-1532-46be-8af1-086a1d709c01" containerName="payload-pre-processing" containerID="cri-o://0fbe58969ada3e96ae961eadcd582b3a5136cbfd702980e2ea8994015a43cf3f" gracePeriod=25
Oct 30 00:24:29.203828 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.203481    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"
Oct 30 00:24:29.322731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.322630    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea8774a1-1532-46be-8af1-086a1d709c01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8774a1-1532-46be-8af1-086a1d709c01-plugins-config-volume\") pod \"ea8774a1-1532-46be-8af1-086a1d709c01\" (UID: \"ea8774a1-1532-46be-8af1-086a1d709c01\") "
Oct 30 00:24:29.323321 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.323289    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8774a1-1532-46be-8af1-086a1d709c01-kube-api-access-7mbgp\" (UniqueName: \"kubernetes.io/projected/ea8774a1-1532-46be-8af1-086a1d709c01-kube-api-access-7mbgp\") pod \"ea8774a1-1532-46be-8af1-086a1d709c01\" (UID: \"ea8774a1-1532-46be-8af1-086a1d709c01\") "
Oct 30 00:24:29.323698 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.323661    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8774a1-1532-46be-8af1-086a1d709c01-plugins-config-volume" pod "ea8774a1-1532-46be-8af1-086a1d709c01" (UID: "ea8774a1-1532-46be-8af1-086a1d709c01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:24:29.324000 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.323975    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8774a1-1532-46be-8af1-086a1d709c01-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:29.325790 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.325760    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8774a1-1532-46be-8af1-086a1d709c01-kube-api-access-7mbgp" pod "ea8774a1-1532-46be-8af1-086a1d709c01" (UID: "ea8774a1-1532-46be-8af1-086a1d709c01"). InnerVolumeSpecName "kube-api-access-7mbgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:24:29.425747 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.425700    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mbgp\" (UniqueName: \"kubernetes.io/projected/ea8774a1-1532-46be-8af1-086a1d709c01-kube-api-access-7mbgp\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:24:29.429424 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.429391    4441 generic.go:356] "Generic (PLEG): container finished" podID="ea8774a1-1532-46be-8af1-086a1d709c01" containerID="0fbe58969ada3e96ae961eadcd582b3a5136cbfd702980e2ea8994015a43cf3f" exitCode=0
Oct 30 00:24:29.429968 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.429453    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"
Oct 30 00:24:29.429968 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.429491    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6" event={"ID":"ea8774a1-1532-46be-8af1-086a1d709c01","Type":"ContainerDied","Data":"0fbe58969ada3e96ae961eadcd582b3a5136cbfd702980e2ea8994015a43cf3f"}
Oct 30 00:24:29.429968 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.429626    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6" event={"ID":"ea8774a1-1532-46be-8af1-086a1d709c01","Type":"ContainerDied","Data":"3b507d29b770a9189698ad025a3b2e514bd2ed0ddb90199bf0efaee1e04a4bc6"}
Oct 30 00:24:29.429968 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.429662    4441 scope.go:122] "RemoveContainer" containerID="0fbe58969ada3e96ae961eadcd582b3a5136cbfd702980e2ea8994015a43cf3f"
Oct 30 00:24:29.442785 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.442763    4441 scope.go:122] "RemoveContainer" containerID="0fbe58969ada3e96ae961eadcd582b3a5136cbfd702980e2ea8994015a43cf3f"
Oct 30 00:24:29.443077 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:24:29.443050    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fbe58969ada3e96ae961eadcd582b3a5136cbfd702980e2ea8994015a43cf3f\": container with ID starting with 0fbe58969ada3e96ae961eadcd582b3a5136cbfd702980e2ea8994015a43cf3f not found: ID does not exist" containerID="0fbe58969ada3e96ae961eadcd582b3a5136cbfd702980e2ea8994015a43cf3f"
Oct 30 00:24:29.443164 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.443085    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fbe58969ada3e96ae961eadcd582b3a5136cbfd702980e2ea8994015a43cf3f"} err="failed to get container status \"0fbe58969ada3e96ae961eadcd582b3a5136cbfd702980e2ea8994015a43cf3f\": rpc error: code = NotFound desc = could not find container \"0fbe58969ada3e96ae961eadcd582b3a5136cbfd702980e2ea8994015a43cf3f\": container with ID starting with 0fbe58969ada3e96ae961eadcd582b3a5136cbfd702980e2ea8994015a43cf3f not found: ID does not exist"
Oct 30 00:24:29.476857 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.476827    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"]
Oct 30 00:24:29.480885 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:29.480859    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5694fc8947-jktl6"]
Oct 30 00:24:30.610475 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:30.610439    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea8774a1-1532-46be-8af1-086a1d709c01" path="/var/lib/kubelet/pods/ea8774a1-1532-46be-8af1-086a1d709c01/volumes"
Oct 30 00:24:32.116622 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.116588    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"]
Oct 30 00:24:32.117371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117343    4441 cpu_manager.go:414] "containerMap: removing container" podUID="ea8774a1-1532-46be-8af1-086a1d709c01" containerName="payload-pre-processing"
Oct 30 00:24:32.117371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117368    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8774a1-1532-46be-8af1-086a1d709c01" containerName="payload-pre-processing"
Oct 30 00:24:32.117556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117394    4441 cpu_manager.go:414] "containerMap: removing container" podUID="71d7723e-9a7d-4858-8093-4b13593809db" containerName="istio-proxy"
Oct 30 00:24:32.117556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117402    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d7723e-9a7d-4858-8093-4b13593809db" containerName="istio-proxy"
Oct 30 00:24:32.117556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117430    4441 cpu_manager.go:414] "containerMap: removing container" podUID="32724ded-d541-4e35-bd55-58b781767acb" containerName="maas-api"
Oct 30 00:24:32.117556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117436    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32724ded-d541-4e35-bd55-58b781767acb" containerName="maas-api"
Oct 30 00:24:32.117556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117449    4441 cpu_manager.go:414] "containerMap: removing container" podUID="96327c70-a528-47d6-ac5d-d68072291d6d" containerName="payload-pre-processing"
Oct 30 00:24:32.117556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117454    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96327c70-a528-47d6-ac5d-d68072291d6d" containerName="payload-pre-processing"
Oct 30 00:24:32.117556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117466    4441 cpu_manager.go:414] "containerMap: removing container" podUID="71d7723e-9a7d-4858-8093-4b13593809db" containerName="istio-proxy"
Oct 30 00:24:32.117556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117471    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d7723e-9a7d-4858-8093-4b13593809db" containerName="istio-proxy"
Oct 30 00:24:32.117556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117490    4441 cpu_manager.go:414] "containerMap: removing container" podUID="caaf8af0-c2b6-4544-896e-fc9cf76e0e99" containerName="payload-processing"
Oct 30 00:24:32.117556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117496    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caaf8af0-c2b6-4544-896e-fc9cf76e0e99" containerName="payload-processing"
Oct 30 00:24:32.117889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117628    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d7723e-9a7d-4858-8093-4b13593809db" containerName="istio-proxy"
Oct 30 00:24:32.117889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117643    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="caaf8af0-c2b6-4544-896e-fc9cf76e0e99" containerName="payload-processing"
Oct 30 00:24:32.117889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117654    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8774a1-1532-46be-8af1-086a1d709c01" containerName="payload-pre-processing"
Oct 30 00:24:32.117889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117663    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="32724ded-d541-4e35-bd55-58b781767acb" containerName="maas-api"
Oct 30 00:24:32.117889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.117671    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="96327c70-a528-47d6-ac5d-d68072291d6d" containerName="payload-pre-processing"
Oct 30 00:24:32.122753 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.122733    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.145329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.145302    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"]
Oct 30 00:24:32.275305 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.275267    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/192d192e-cab7-42b8-86d3-335abcced580-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.275483 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.275448    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxjx\" (UniqueName: \"kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-kube-api-access-6bxjx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.275540 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.275507    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.275683 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.275660    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.275802 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.275784    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.275878 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.275831    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.275936 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.275907    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.377665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.377580    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/192d192e-cab7-42b8-86d3-335abcced580-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.377831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.377713    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxjx\" (UniqueName: \"kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-kube-api-access-6bxjx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.377831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.377771    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.377981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.377956    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.378110 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.378094    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.378193 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.378127    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.378193 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.378152    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.378896 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.378873    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.380100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.380075    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/192d192e-cab7-42b8-86d3-335abcced580-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.380196 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.380172    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.380616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.380600    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.380665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.380620    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.401597 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.401565    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxjx\" (UniqueName: \"kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-kube-api-access-6bxjx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.401776 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.401757    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bv72k\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.445392 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.442431    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:32.819164 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.819133    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"]
Oct 30 00:24:32.819359 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:24:32.819217    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192d192e_cab7_42b8_86d3_335abcced580.slice/crio-1dd1cd643b30c5c9bda67c90606d9d96b9baaaec7c4ce8b40ecf3b483afa87ab WatchSource:0}: Error finding container 1dd1cd643b30c5c9bda67c90606d9d96b9baaaec7c4ce8b40ecf3b483afa87ab: Status 404 returned error can't find the container with id 1dd1cd643b30c5c9bda67c90606d9d96b9baaaec7c4ce8b40ecf3b483afa87ab
Oct 30 00:24:32.822106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.822065    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:24:32.822222 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:32.822132    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:24:33.481706 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:33.481656    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k" event={"ID":"192d192e-cab7-42b8-86d3-335abcced580","Type":"ContainerStarted","Data":"eb7b71f1494879e229da0246d7e694c024ae81aa8386a3d51f273df0dddefe8c"}
Oct 30 00:24:33.482832 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:33.481725    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k" event={"ID":"192d192e-cab7-42b8-86d3-335abcced580","Type":"ContainerStarted","Data":"1dd1cd643b30c5c9bda67c90606d9d96b9baaaec7c4ce8b40ecf3b483afa87ab"}
Oct 30 00:24:33.482832 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:33.482528    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:33.484829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:33.484361    4441 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bv72k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:24:33.484829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:33.484433    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k" podUID="192d192e-cab7-42b8-86d3-335abcced580" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:24:33.552392 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:33.552321    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k" podStartSLOduration=1.552300617 podStartE2EDuration="1.552300617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:33.532676785 +0000 UTC m=+1183.057970663" watchObservedRunningTime="2026-10-30 00:24:33.552300617 +0000 UTC m=+1183.077594495"
Oct 30 00:24:33.906524 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:24:33.906429    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:24:34.489499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:34.489472    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:24:56.805107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:56.805077    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:24:56.813018 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:24:56.812995    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:25:03.903827 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:03.903774    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:25:05.138430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:05.138388    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"]
Oct 30 00:25:05.242453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:05.242420    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"]
Oct 30 00:25:08.372789 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:08.372754    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"]
Oct 30 00:25:08.373249 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:08.372984    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s" podUID="0360d680-dc02-4fd2-a4fd-47f584e20527" containerName="istio-proxy" containerID="cri-o://4ff805a1c6fa32a453fd2739f4498fd8a9251d57189d439530ebfa5fcacc7f12" gracePeriod=30
Oct 30 00:25:08.377679 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:08.377650    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"]
Oct 30 00:25:08.377905 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:08.377863    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g" podUID="dffac52a-994f-43ce-ad5b-0abf6a73f13a" containerName="istio-proxy" containerID="cri-o://36e832d24369d88e6e1dd68d3f0c57ade14751b70188ec1553b045720a9bb97a" gracePeriod=30
Oct 30 00:25:08.381753 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:08.381728    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"]
Oct 30 00:25:08.381937 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:08.381916    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5" podUID="0298952c-f699-4f6b-b32a-f35db6d0e113" containerName="istio-proxy" containerID="cri-o://f53d85cfd3fd9df994d8416d38d9143eae10b778b1ec1a62cc6ff849233c735e" gracePeriod=30
Oct 30 00:25:10.139744 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.139695    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq" podUID="65f4a9c0-bdd6-45d7-b363-f64e7461530c" containerName="payload-processing" containerID="cri-o://e0fca206ead763d19ebf62f23d3cb25e9b317ea58a22061051114426e35f2c01" gracePeriod=25
Oct 30 00:25:10.243396 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.243185    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n" podUID="350647e3-09d0-481a-baaf-dc0043c75ba7" containerName="payload-pre-processing" containerID="cri-o://58ec3805eaee627c2e9cb1ef2d70a825b303a16ff22d572c7037629c9cd4c4e8" gracePeriod=25
Oct 30 00:25:10.653476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.650669    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"
Oct 30 00:25:10.690490 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.690458    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"
Oct 30 00:25:10.754856 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.754815    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f4a9c0-bdd6-45d7-b363-f64e7461530c-kube-api-access-7xz6k\" (UniqueName: \"kubernetes.io/projected/65f4a9c0-bdd6-45d7-b363-f64e7461530c-kube-api-access-7xz6k\") pod \"65f4a9c0-bdd6-45d7-b363-f64e7461530c\" (UID: \"65f4a9c0-bdd6-45d7-b363-f64e7461530c\") "
Oct 30 00:25:10.755125 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.755102    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f4a9c0-bdd6-45d7-b363-f64e7461530c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f4a9c0-bdd6-45d7-b363-f64e7461530c-plugins-config-volume\") pod \"65f4a9c0-bdd6-45d7-b363-f64e7461530c\" (UID: \"65f4a9c0-bdd6-45d7-b363-f64e7461530c\") "
Oct 30 00:25:10.755222 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.755167    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/350647e3-09d0-481a-baaf-dc0043c75ba7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350647e3-09d0-481a-baaf-dc0043c75ba7-plugins-config-volume\") pod \"350647e3-09d0-481a-baaf-dc0043c75ba7\" (UID: \"350647e3-09d0-481a-baaf-dc0043c75ba7\") "
Oct 30 00:25:10.755222 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.755211    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350647e3-09d0-481a-baaf-dc0043c75ba7-kube-api-access-4lngb\" (UniqueName: \"kubernetes.io/projected/350647e3-09d0-481a-baaf-dc0043c75ba7-kube-api-access-4lngb\") pod \"350647e3-09d0-481a-baaf-dc0043c75ba7\" (UID: \"350647e3-09d0-481a-baaf-dc0043c75ba7\") "
Oct 30 00:25:10.757098 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.757067    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/350647e3-09d0-481a-baaf-dc0043c75ba7-plugins-config-volume" pod "350647e3-09d0-481a-baaf-dc0043c75ba7" (UID: "350647e3-09d0-481a-baaf-dc0043c75ba7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:25:10.757497 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.757474    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f4a9c0-bdd6-45d7-b363-f64e7461530c-plugins-config-volume" pod "65f4a9c0-bdd6-45d7-b363-f64e7461530c" (UID: "65f4a9c0-bdd6-45d7-b363-f64e7461530c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:25:10.763595 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.763565    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350647e3-09d0-481a-baaf-dc0043c75ba7-kube-api-access-4lngb" pod "350647e3-09d0-481a-baaf-dc0043c75ba7" (UID: "350647e3-09d0-481a-baaf-dc0043c75ba7"). InnerVolumeSpecName "kube-api-access-4lngb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:10.765484 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.765450    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f4a9c0-bdd6-45d7-b363-f64e7461530c-kube-api-access-7xz6k" pod "65f4a9c0-bdd6-45d7-b363-f64e7461530c" (UID: "65f4a9c0-bdd6-45d7-b363-f64e7461530c"). InnerVolumeSpecName "kube-api-access-7xz6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:10.858269 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.858225    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f4a9c0-bdd6-45d7-b363-f64e7461530c-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:10.858269 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.858262    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350647e3-09d0-481a-baaf-dc0043c75ba7-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:10.858269 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.858277    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lngb\" (UniqueName: \"kubernetes.io/projected/350647e3-09d0-481a-baaf-dc0043c75ba7-kube-api-access-4lngb\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:10.858575 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:10.858292    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xz6k\" (UniqueName: \"kubernetes.io/projected/65f4a9c0-bdd6-45d7-b363-f64e7461530c-kube-api-access-7xz6k\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:11.125609 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.125499    4441 generic.go:356] "Generic (PLEG): container finished" podID="65f4a9c0-bdd6-45d7-b363-f64e7461530c" containerID="e0fca206ead763d19ebf62f23d3cb25e9b317ea58a22061051114426e35f2c01" exitCode=0
Oct 30 00:25:11.125609 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.125554    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"
Oct 30 00:25:11.125609 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.125579    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq" event={"ID":"65f4a9c0-bdd6-45d7-b363-f64e7461530c","Type":"ContainerDied","Data":"e0fca206ead763d19ebf62f23d3cb25e9b317ea58a22061051114426e35f2c01"}
Oct 30 00:25:11.125922 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.125673    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq" event={"ID":"65f4a9c0-bdd6-45d7-b363-f64e7461530c","Type":"ContainerDied","Data":"ed4d14a53f9ce6d795cfff0e11a894955a1af0199217b7cb4e07664215cf9bf8"}
Oct 30 00:25:11.125922 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.125708    4441 scope.go:122] "RemoveContainer" containerID="e0fca206ead763d19ebf62f23d3cb25e9b317ea58a22061051114426e35f2c01"
Oct 30 00:25:11.127258 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.127234    4441 generic.go:356] "Generic (PLEG): container finished" podID="350647e3-09d0-481a-baaf-dc0043c75ba7" containerID="58ec3805eaee627c2e9cb1ef2d70a825b303a16ff22d572c7037629c9cd4c4e8" exitCode=0
Oct 30 00:25:11.127466 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.127450    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n" event={"ID":"350647e3-09d0-481a-baaf-dc0043c75ba7","Type":"ContainerDied","Data":"58ec3805eaee627c2e9cb1ef2d70a825b303a16ff22d572c7037629c9cd4c4e8"}
Oct 30 00:25:11.127538 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.127493    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n" event={"ID":"350647e3-09d0-481a-baaf-dc0043c75ba7","Type":"ContainerDied","Data":"a59cddd9cb26261218bb141c7461dfdc1cf3afd0bafaec93b9cb559d2316c6b8"}
Oct 30 00:25:11.127611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.127594    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"
Oct 30 00:25:11.168541 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.168515    4441 scope.go:122] "RemoveContainer" containerID="e0fca206ead763d19ebf62f23d3cb25e9b317ea58a22061051114426e35f2c01"
Oct 30 00:25:11.168930 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:11.168848    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0fca206ead763d19ebf62f23d3cb25e9b317ea58a22061051114426e35f2c01\": container with ID starting with e0fca206ead763d19ebf62f23d3cb25e9b317ea58a22061051114426e35f2c01 not found: ID does not exist" containerID="e0fca206ead763d19ebf62f23d3cb25e9b317ea58a22061051114426e35f2c01"
Oct 30 00:25:11.168930 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.168883    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fca206ead763d19ebf62f23d3cb25e9b317ea58a22061051114426e35f2c01"} err="failed to get container status \"e0fca206ead763d19ebf62f23d3cb25e9b317ea58a22061051114426e35f2c01\": rpc error: code = NotFound desc = could not find container \"e0fca206ead763d19ebf62f23d3cb25e9b317ea58a22061051114426e35f2c01\": container with ID starting with e0fca206ead763d19ebf62f23d3cb25e9b317ea58a22061051114426e35f2c01 not found: ID does not exist"
Oct 30 00:25:11.168930 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.168906    4441 scope.go:122] "RemoveContainer" containerID="58ec3805eaee627c2e9cb1ef2d70a825b303a16ff22d572c7037629c9cd4c4e8"
Oct 30 00:25:11.183624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.183598    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"]
Oct 30 00:25:11.183958 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.183938    4441 scope.go:122] "RemoveContainer" containerID="58ec3805eaee627c2e9cb1ef2d70a825b303a16ff22d572c7037629c9cd4c4e8"
Oct 30 00:25:11.184392 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:11.184368    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ec3805eaee627c2e9cb1ef2d70a825b303a16ff22d572c7037629c9cd4c4e8\": container with ID starting with 58ec3805eaee627c2e9cb1ef2d70a825b303a16ff22d572c7037629c9cd4c4e8 not found: ID does not exist" containerID="58ec3805eaee627c2e9cb1ef2d70a825b303a16ff22d572c7037629c9cd4c4e8"
Oct 30 00:25:11.184468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.184403    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ec3805eaee627c2e9cb1ef2d70a825b303a16ff22d572c7037629c9cd4c4e8"} err="failed to get container status \"58ec3805eaee627c2e9cb1ef2d70a825b303a16ff22d572c7037629c9cd4c4e8\": rpc error: code = NotFound desc = could not find container \"58ec3805eaee627c2e9cb1ef2d70a825b303a16ff22d572c7037629c9cd4c4e8\": container with ID starting with 58ec3805eaee627c2e9cb1ef2d70a825b303a16ff22d572c7037629c9cd4c4e8 not found: ID does not exist"
Oct 30 00:25:11.192989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.192965    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0847e8c0-5747799dd6-cvrgq"]
Oct 30 00:25:11.215537 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.215514    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"]
Oct 30 00:25:11.219347 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.219325    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-f9c8n"]
Oct 30 00:25:11.627958 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.627923    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"]
Oct 30 00:25:11.628204 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:11.628166    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz" podUID="bee7d813-eb28-46ed-b93c-f5422243a68a" containerName="maas-api" containerID="cri-o://ec480bededaebdf53698a1f068604353dfb76daef31b9f68553077162ba8c9a8" gracePeriod=30
Oct 30 00:25:12.059051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.052847    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:25:12.091057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.088208    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee7d813-eb28-46ed-b93c-f5422243a68a-kube-api-access-x89sl\" (UniqueName: \"kubernetes.io/projected/bee7d813-eb28-46ed-b93c-f5422243a68a-kube-api-access-x89sl\") pod \"bee7d813-eb28-46ed-b93c-f5422243a68a\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") "
Oct 30 00:25:12.091057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.088321    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-metrics-certs\") pod \"bee7d813-eb28-46ed-b93c-f5422243a68a\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") "
Oct 30 00:25:12.091057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.088670    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee7d813-eb28-46ed-b93c-f5422243a68a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee7d813-eb28-46ed-b93c-f5422243a68a-openshift-service-ca\") pod \"bee7d813-eb28-46ed-b93c-f5422243a68a\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") "
Oct 30 00:25:12.091057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.088750    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-maas-api-tls\") pod \"bee7d813-eb28-46ed-b93c-f5422243a68a\" (UID: \"bee7d813-eb28-46ed-b93c-f5422243a68a\") "
Oct 30 00:25:12.128105 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.114122    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee7d813-eb28-46ed-b93c-f5422243a68a-openshift-service-ca" pod "bee7d813-eb28-46ed-b93c-f5422243a68a" (UID: "bee7d813-eb28-46ed-b93c-f5422243a68a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:25:12.192916 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.191320    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee7d813-eb28-46ed-b93c-f5422243a68a-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:12.232502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.231841    4441 generic.go:356] "Generic (PLEG): container finished" podID="bee7d813-eb28-46ed-b93c-f5422243a68a" containerID="ec480bededaebdf53698a1f068604353dfb76daef31b9f68553077162ba8c9a8" exitCode=0
Oct 30 00:25:12.232502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.231898    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz" event={"ID":"bee7d813-eb28-46ed-b93c-f5422243a68a","Type":"ContainerDied","Data":"ec480bededaebdf53698a1f068604353dfb76daef31b9f68553077162ba8c9a8"}
Oct 30 00:25:12.232502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.231942    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz" event={"ID":"bee7d813-eb28-46ed-b93c-f5422243a68a","Type":"ContainerDied","Data":"6cc1d2e2004032eb15e004b619d6eb103e7370fd75b90d18a7b4694622843c88"}
Oct 30 00:25:12.232502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.231969    4441 scope.go:122] "RemoveContainer" containerID="ec480bededaebdf53698a1f068604353dfb76daef31b9f68553077162ba8c9a8"
Oct 30 00:25:12.232502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.232103    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"
Oct 30 00:25:12.235333 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.233734    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee7d813-eb28-46ed-b93c-f5422243a68a-kube-api-access-x89sl" pod "bee7d813-eb28-46ed-b93c-f5422243a68a" (UID: "bee7d813-eb28-46ed-b93c-f5422243a68a"). InnerVolumeSpecName "kube-api-access-x89sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:12.235333 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.233899    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-maas-api-tls" pod "bee7d813-eb28-46ed-b93c-f5422243a68a" (UID: "bee7d813-eb28-46ed-b93c-f5422243a68a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:25:12.262174 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.257487    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-metrics-certs" pod "bee7d813-eb28-46ed-b93c-f5422243a68a" (UID: "bee7d813-eb28-46ed-b93c-f5422243a68a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:25:12.264485 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.264154    4441 scope.go:122] "RemoveContainer" containerID="ec480bededaebdf53698a1f068604353dfb76daef31b9f68553077162ba8c9a8"
Oct 30 00:25:12.265416 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:12.264955    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec480bededaebdf53698a1f068604353dfb76daef31b9f68553077162ba8c9a8\": container with ID starting with ec480bededaebdf53698a1f068604353dfb76daef31b9f68553077162ba8c9a8 not found: ID does not exist" containerID="ec480bededaebdf53698a1f068604353dfb76daef31b9f68553077162ba8c9a8"
Oct 30 00:25:12.265416 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.264994    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec480bededaebdf53698a1f068604353dfb76daef31b9f68553077162ba8c9a8"} err="failed to get container status \"ec480bededaebdf53698a1f068604353dfb76daef31b9f68553077162ba8c9a8\": rpc error: code = NotFound desc = could not find container \"ec480bededaebdf53698a1f068604353dfb76daef31b9f68553077162ba8c9a8\": container with ID starting with ec480bededaebdf53698a1f068604353dfb76daef31b9f68553077162ba8c9a8 not found: ID does not exist"
Oct 30 00:25:12.293788 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.293710    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:12.293788 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.293749    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x89sl\" (UniqueName: \"kubernetes.io/projected/bee7d813-eb28-46ed-b93c-f5422243a68a-kube-api-access-x89sl\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:12.293788 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.293764    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bee7d813-eb28-46ed-b93c-f5422243a68a-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:12.579563 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.579532    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"]
Oct 30 00:25:12.588821 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.588795    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c34304-77dc7c558d-pfvfz"]
Oct 30 00:25:12.614171 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.614136    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="350647e3-09d0-481a-baaf-dc0043c75ba7" path="/var/lib/kubelet/pods/350647e3-09d0-481a-baaf-dc0043c75ba7/volumes"
Oct 30 00:25:12.614711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.614687    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f4a9c0-bdd6-45d7-b363-f64e7461530c" path="/var/lib/kubelet/pods/65f4a9c0-bdd6-45d7-b363-f64e7461530c/volumes"
Oct 30 00:25:12.615396 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:12.615373    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee7d813-eb28-46ed-b93c-f5422243a68a" path="/var/lib/kubelet/pods/bee7d813-eb28-46ed-b93c-f5422243a68a/volumes"
Oct 30 00:25:13.549060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:13.545963    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"]
Oct 30 00:25:13.549060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:13.546238    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57" podUID="f9131a7c-3280-43e8-a388-e9c51a7a0389" containerName="maas-api" containerID="cri-o://785921c5788f789d12126ab325e03f2b22906183258a336dce129db25e5fcae1" gracePeriod=30
Oct 30 00:25:13.977103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:13.976731    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:25:13.978813 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:13.978780    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:25:13.994827 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:13.994704    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:25:14.023372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023337    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-envoy\") pod \"0360d680-dc02-4fd2-a4fd-47f584e20527\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") "
Oct 30 00:25:14.023542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023413    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istiod-ca-cert\") pod \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") "
Oct 30 00:25:14.023542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023464    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-socket\") pod \"0360d680-dc02-4fd2-a4fd-47f584e20527\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") "
Oct 30 00:25:14.023542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023490    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0360d680-dc02-4fd2-a4fd-47f584e20527-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0360d680-dc02-4fd2-a4fd-47f584e20527-istiod-ca-cert\") pod \"0360d680-dc02-4fd2-a4fd-47f584e20527\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") "
Oct 30 00:25:14.023542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023531    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-certs\") pod \"0298952c-f699-4f6b-b32a-f35db6d0e113\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") "
Oct 30 00:25:14.023907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023595    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-kube-api-access-cmzv5\" (UniqueName: \"kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-kube-api-access-cmzv5\") pod \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") "
Oct 30 00:25:14.023907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023628    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0298952c-f699-4f6b-b32a-f35db6d0e113-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0298952c-f699-4f6b-b32a-f35db6d0e113-istiod-ca-cert\") pod \"0298952c-f699-4f6b-b32a-f35db6d0e113\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") "
Oct 30 00:25:14.023907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023658    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-socket\") pod \"0298952c-f699-4f6b-b32a-f35db6d0e113\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") "
Oct 30 00:25:14.023907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023691    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-credential-socket\") pod \"0360d680-dc02-4fd2-a4fd-47f584e20527\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") "
Oct 30 00:25:14.023907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023765    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-envoy\") pod \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") "
Oct 30 00:25:14.023907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023832    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-token\" (UniqueName: \"kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-token\") pod \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") "
Oct 30 00:25:14.023907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023868    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-token\" (UniqueName: \"kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-token\") pod \"0298952c-f699-4f6b-b32a-f35db6d0e113\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") "
Oct 30 00:25:14.023907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023898    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-data\") pod \"0298952c-f699-4f6b-b32a-f35db6d0e113\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") "
Oct 30 00:25:14.024356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023953    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-envoy\") pod \"0298952c-f699-4f6b-b32a-f35db6d0e113\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") "
Oct 30 00:25:14.024356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.023979    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-credential-socket\") pod \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") "
Oct 30 00:25:14.024356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.024038    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-data\") pod \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") "
Oct 30 00:25:14.024356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.024082    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-socket\") pod \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") "
Oct 30 00:25:14.024356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.024183    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-certs\") pod \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") "
Oct 30 00:25:14.024356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.024282    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-kube-api-access-cnn8n\" (UniqueName: \"kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-kube-api-access-cnn8n\") pod \"0360d680-dc02-4fd2-a4fd-47f584e20527\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") "
Oct 30 00:25:14.024356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.024328    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-data\") pod \"0360d680-dc02-4fd2-a4fd-47f584e20527\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") "
Oct 30 00:25:14.024696 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.024375    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-token\" (UniqueName: \"kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-token\") pod \"0360d680-dc02-4fd2-a4fd-47f584e20527\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") "
Oct 30 00:25:14.024696 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.024423    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-certs\") pod \"0360d680-dc02-4fd2-a4fd-47f584e20527\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") "
Oct 30 00:25:14.025624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.025485    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-podinfo\") pod \"0360d680-dc02-4fd2-a4fd-47f584e20527\" (UID: \"0360d680-dc02-4fd2-a4fd-47f584e20527\") "
Oct 30 00:25:14.025624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.025557    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-kube-api-access-hbk99\" (UniqueName: \"kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-kube-api-access-hbk99\") pod \"0298952c-f699-4f6b-b32a-f35db6d0e113\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") "
Oct 30 00:25:14.026423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.025994    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-credential-socket\") pod \"0298952c-f699-4f6b-b32a-f35db6d0e113\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") "
Oct 30 00:25:14.026423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.026079    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-podinfo\") pod \"0298952c-f699-4f6b-b32a-f35db6d0e113\" (UID: \"0298952c-f699-4f6b-b32a-f35db6d0e113\") "
Oct 30 00:25:14.026423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.026141    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-podinfo\") pod \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\" (UID: \"dffac52a-994f-43ce-ad5b-0abf6a73f13a\") "
Oct 30 00:25:14.028764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.028264    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-certs" pod "dffac52a-994f-43ce-ad5b-0abf6a73f13a" (UID: "dffac52a-994f-43ce-ad5b-0abf6a73f13a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.028764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.028511    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-data" pod "0298952c-f699-4f6b-b32a-f35db6d0e113" (UID: "0298952c-f699-4f6b-b32a-f35db6d0e113"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.028930 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.028836    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-credential-socket" pod "0298952c-f699-4f6b-b32a-f35db6d0e113" (UID: "0298952c-f699-4f6b-b32a-f35db6d0e113"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.031168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.031056    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-certs" pod "0360d680-dc02-4fd2-a4fd-47f584e20527" (UID: "0360d680-dc02-4fd2-a4fd-47f584e20527"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.032276 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.031498    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0360d680-dc02-4fd2-a4fd-47f584e20527-istiod-ca-cert" pod "0360d680-dc02-4fd2-a4fd-47f584e20527" (UID: "0360d680-dc02-4fd2-a4fd-47f584e20527"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:25:14.032467 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.032369    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-socket" pod "0360d680-dc02-4fd2-a4fd-47f584e20527" (UID: "0360d680-dc02-4fd2-a4fd-47f584e20527"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.032590 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.032568    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-credential-socket" pod "0360d680-dc02-4fd2-a4fd-47f584e20527" (UID: "0360d680-dc02-4fd2-a4fd-47f584e20527"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.032771 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.032744    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istiod-ca-cert" pod "dffac52a-994f-43ce-ad5b-0abf6a73f13a" (UID: "dffac52a-994f-43ce-ad5b-0abf6a73f13a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:25:14.032851 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.032809    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-certs" pod "0298952c-f699-4f6b-b32a-f35db6d0e113" (UID: "0298952c-f699-4f6b-b32a-f35db6d0e113"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.032940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.032875    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0298952c-f699-4f6b-b32a-f35db6d0e113-istiod-ca-cert" pod "0298952c-f699-4f6b-b32a-f35db6d0e113" (UID: "0298952c-f699-4f6b-b32a-f35db6d0e113"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:25:14.039366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.033110    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-data" pod "0360d680-dc02-4fd2-a4fd-47f584e20527" (UID: "0360d680-dc02-4fd2-a4fd-47f584e20527"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.039366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.033167    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-socket" pod "0298952c-f699-4f6b-b32a-f35db6d0e113" (UID: "0298952c-f699-4f6b-b32a-f35db6d0e113"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.039366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.037051    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-socket" pod "dffac52a-994f-43ce-ad5b-0abf6a73f13a" (UID: "dffac52a-994f-43ce-ad5b-0abf6a73f13a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.039366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.037060    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-data" pod "dffac52a-994f-43ce-ad5b-0abf6a73f13a" (UID: "dffac52a-994f-43ce-ad5b-0abf6a73f13a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.039366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.037243    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-credential-socket" pod "dffac52a-994f-43ce-ad5b-0abf6a73f13a" (UID: "dffac52a-994f-43ce-ad5b-0abf6a73f13a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.049289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.045254    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-podinfo" pod "dffac52a-994f-43ce-ad5b-0abf6a73f13a" (UID: "dffac52a-994f-43ce-ad5b-0abf6a73f13a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:25:14.049289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.045296    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-token" pod "0298952c-f699-4f6b-b32a-f35db6d0e113" (UID: "0298952c-f699-4f6b-b32a-f35db6d0e113"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:14.049289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.045356    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-envoy" pod "0360d680-dc02-4fd2-a4fd-47f584e20527" (UID: "0360d680-dc02-4fd2-a4fd-47f584e20527"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.049289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.045792    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-kube-api-access-cnn8n" pod "0360d680-dc02-4fd2-a4fd-47f584e20527" (UID: "0360d680-dc02-4fd2-a4fd-47f584e20527"). InnerVolumeSpecName "kube-api-access-cnn8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:14.049289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.045913    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-podinfo" pod "0360d680-dc02-4fd2-a4fd-47f584e20527" (UID: "0360d680-dc02-4fd2-a4fd-47f584e20527"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:25:14.049289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.045923    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-podinfo" pod "0298952c-f699-4f6b-b32a-f35db6d0e113" (UID: "0298952c-f699-4f6b-b32a-f35db6d0e113"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:25:14.049289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.045978    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-token" pod "0360d680-dc02-4fd2-a4fd-47f584e20527" (UID: "0360d680-dc02-4fd2-a4fd-47f584e20527"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:14.052202 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.046013    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-envoy" pod "0298952c-f699-4f6b-b32a-f35db6d0e113" (UID: "0298952c-f699-4f6b-b32a-f35db6d0e113"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.052202 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.046058    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-kube-api-access-cmzv5" pod "dffac52a-994f-43ce-ad5b-0abf6a73f13a" (UID: "dffac52a-994f-43ce-ad5b-0abf6a73f13a"). InnerVolumeSpecName "kube-api-access-cmzv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:14.052202 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.046106    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-token" pod "dffac52a-994f-43ce-ad5b-0abf6a73f13a" (UID: "dffac52a-994f-43ce-ad5b-0abf6a73f13a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:14.052202 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.046696    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-kube-api-access-hbk99" pod "0298952c-f699-4f6b-b32a-f35db6d0e113" (UID: "0298952c-f699-4f6b-b32a-f35db6d0e113"). InnerVolumeSpecName "kube-api-access-hbk99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:14.057201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.057140    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-envoy" pod "dffac52a-994f-43ce-ad5b-0abf6a73f13a" (UID: "dffac52a-994f-43ce-ad5b-0abf6a73f13a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:14.060254 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.060233    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:25:14.177392 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.177308    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9131a7c-3280-43e8-a388-e9c51a7a0389-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9131a7c-3280-43e8-a388-e9c51a7a0389-openshift-service-ca\") pod \"f9131a7c-3280-43e8-a388-e9c51a7a0389\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") "
Oct 30 00:25:14.178042 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.178005    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-maas-api-tls\") pod \"f9131a7c-3280-43e8-a388-e9c51a7a0389\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") "
Oct 30 00:25:14.178302 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.178283    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9131a7c-3280-43e8-a388-e9c51a7a0389-kube-api-access-7xggl\" (UniqueName: \"kubernetes.io/projected/f9131a7c-3280-43e8-a388-e9c51a7a0389-kube-api-access-7xggl\") pod \"f9131a7c-3280-43e8-a388-e9c51a7a0389\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") "
Oct 30 00:25:14.178484 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.178470    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-metrics-certs\") pod \"f9131a7c-3280-43e8-a388-e9c51a7a0389\" (UID: \"f9131a7c-3280-43e8-a388-e9c51a7a0389\") "
Oct 30 00:25:14.179652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.179626    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.179790 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.179776    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnn8n\" (UniqueName: \"kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-kube-api-access-cnn8n\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.179892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.179879    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.179975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.179964    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.183421 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.183405    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.183560 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.183548    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.183655 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.183644    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbk99\" (UniqueName: \"kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-kube-api-access-hbk99\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.183748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.183736    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.183833 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.183823    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.183922 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.183912    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184019 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184008    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184133 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.177894    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9131a7c-3280-43e8-a388-e9c51a7a0389-openshift-service-ca" pod "f9131a7c-3280-43e8-a388-e9c51a7a0389" (UID: "f9131a7c-3280-43e8-a388-e9c51a7a0389"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:25:14.184237 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184120    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184300 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184249    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184300 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184268    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0360d680-dc02-4fd2-a4fd-47f584e20527-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184300 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184284    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184297    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmzv5\" (UniqueName: \"kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-kube-api-access-cmzv5\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184314    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0298952c-f699-4f6b-b32a-f35db6d0e113-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184327    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184341    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0360d680-dc02-4fd2-a4fd-47f584e20527-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184354    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184376    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184388    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184400    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184413    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0298952c-f699-4f6b-b32a-f35db6d0e113-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184425    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184437    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.184470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.184450    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffac52a-994f-43ce-ad5b-0abf6a73f13a-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.201089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.193710    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-maas-api-tls" pod "f9131a7c-3280-43e8-a388-e9c51a7a0389" (UID: "f9131a7c-3280-43e8-a388-e9c51a7a0389"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:25:14.201089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.195529    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9131a7c-3280-43e8-a388-e9c51a7a0389-kube-api-access-7xggl" pod "f9131a7c-3280-43e8-a388-e9c51a7a0389" (UID: "f9131a7c-3280-43e8-a388-e9c51a7a0389"). InnerVolumeSpecName "kube-api-access-7xggl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:14.201089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.195746    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-metrics-certs" pod "f9131a7c-3280-43e8-a388-e9c51a7a0389" (UID: "f9131a7c-3280-43e8-a388-e9c51a7a0389"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:25:14.256955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.256915    4441 generic.go:356] "Generic (PLEG): container finished" podID="dffac52a-994f-43ce-ad5b-0abf6a73f13a" containerID="36e832d24369d88e6e1dd68d3f0c57ade14751b70188ec1553b045720a9bb97a" exitCode=0
Oct 30 00:25:14.257301 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.257275    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g" event={"ID":"dffac52a-994f-43ce-ad5b-0abf6a73f13a","Type":"ContainerDied","Data":"36e832d24369d88e6e1dd68d3f0c57ade14751b70188ec1553b045720a9bb97a"}
Oct 30 00:25:14.257539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.257502    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g" event={"ID":"dffac52a-994f-43ce-ad5b-0abf6a73f13a","Type":"ContainerDied","Data":"b85af8595aa7cf7183202bee6d9495d6acf1d5ba862a460a6e44c306db0d8877"}
Oct 30 00:25:14.257711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.257697    4441 scope.go:122] "RemoveContainer" containerID="36e832d24369d88e6e1dd68d3f0c57ade14751b70188ec1553b045720a9bb97a"
Oct 30 00:25:14.258014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.257999    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"
Oct 30 00:25:14.261398 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.261374    4441 generic.go:356] "Generic (PLEG): container finished" podID="f9131a7c-3280-43e8-a388-e9c51a7a0389" containerID="785921c5788f789d12126ab325e03f2b22906183258a336dce129db25e5fcae1" exitCode=0
Oct 30 00:25:14.261785 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.261750    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57" event={"ID":"f9131a7c-3280-43e8-a388-e9c51a7a0389","Type":"ContainerDied","Data":"785921c5788f789d12126ab325e03f2b22906183258a336dce129db25e5fcae1"}
Oct 30 00:25:14.261942 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.261922    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57" event={"ID":"f9131a7c-3280-43e8-a388-e9c51a7a0389","Type":"ContainerDied","Data":"611ee5feca68607ecfbe28f280a9ef6e0de06a21c2fb86f62a2dc3187ee087f8"}
Oct 30 00:25:14.270469 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.270448    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"
Oct 30 00:25:14.286176 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.286148    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9131a7c-3280-43e8-a388-e9c51a7a0389-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.286306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.286180    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.286306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.286194    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xggl\" (UniqueName: \"kubernetes.io/projected/f9131a7c-3280-43e8-a388-e9c51a7a0389-kube-api-access-7xggl\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.286306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.286205    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9131a7c-3280-43e8-a388-e9c51a7a0389-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:14.288927 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.288881    4441 generic.go:356] "Generic (PLEG): container finished" podID="0360d680-dc02-4fd2-a4fd-47f584e20527" containerID="4ff805a1c6fa32a453fd2739f4498fd8a9251d57189d439530ebfa5fcacc7f12" exitCode=0
Oct 30 00:25:14.289064 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.289003    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s" event={"ID":"0360d680-dc02-4fd2-a4fd-47f584e20527","Type":"ContainerDied","Data":"4ff805a1c6fa32a453fd2739f4498fd8a9251d57189d439530ebfa5fcacc7f12"}
Oct 30 00:25:14.289133 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.289081    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s" event={"ID":"0360d680-dc02-4fd2-a4fd-47f584e20527","Type":"ContainerDied","Data":"97b6cb72842818af231722599239156c3ad14e816545fa1125ada821ee8c7c61"}
Oct 30 00:25:14.289283 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.289265    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"
Oct 30 00:25:14.294934 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.294904    4441 generic.go:356] "Generic (PLEG): container finished" podID="0298952c-f699-4f6b-b32a-f35db6d0e113" containerID="f53d85cfd3fd9df994d8416d38d9143eae10b778b1ec1a62cc6ff849233c735e" exitCode=0
Oct 30 00:25:14.295072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.294990    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5" event={"ID":"0298952c-f699-4f6b-b32a-f35db6d0e113","Type":"ContainerDied","Data":"f53d85cfd3fd9df994d8416d38d9143eae10b778b1ec1a62cc6ff849233c735e"}
Oct 30 00:25:14.295072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.295051    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5" event={"ID":"0298952c-f699-4f6b-b32a-f35db6d0e113","Type":"ContainerDied","Data":"ad313330a4e12e8e4acd731bffcb08d72118a578ec88af7c34086f23fa649a27"}
Oct 30 00:25:14.295201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.295146    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"
Oct 30 00:25:14.303462 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.303436    4441 scope.go:122] "RemoveContainer" containerID="36e832d24369d88e6e1dd68d3f0c57ade14751b70188ec1553b045720a9bb97a"
Oct 30 00:25:14.305591 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:14.305561    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e832d24369d88e6e1dd68d3f0c57ade14751b70188ec1553b045720a9bb97a\": container with ID starting with 36e832d24369d88e6e1dd68d3f0c57ade14751b70188ec1553b045720a9bb97a not found: ID does not exist" containerID="36e832d24369d88e6e1dd68d3f0c57ade14751b70188ec1553b045720a9bb97a"
Oct 30 00:25:14.305707 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.305603    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e832d24369d88e6e1dd68d3f0c57ade14751b70188ec1553b045720a9bb97a"} err="failed to get container status \"36e832d24369d88e6e1dd68d3f0c57ade14751b70188ec1553b045720a9bb97a\": rpc error: code = NotFound desc = could not find container \"36e832d24369d88e6e1dd68d3f0c57ade14751b70188ec1553b045720a9bb97a\": container with ID starting with 36e832d24369d88e6e1dd68d3f0c57ade14751b70188ec1553b045720a9bb97a not found: ID does not exist"
Oct 30 00:25:14.305707 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.305627    4441 scope.go:122] "RemoveContainer" containerID="785921c5788f789d12126ab325e03f2b22906183258a336dce129db25e5fcae1"
Oct 30 00:25:14.370051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.360861    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"]
Oct 30 00:25:14.370051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.369476    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0847e8c0-585888c668-lrz57"]
Oct 30 00:25:14.372087 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.372058    4441 scope.go:122] "RemoveContainer" containerID="785921c5788f789d12126ab325e03f2b22906183258a336dce129db25e5fcae1"
Oct 30 00:25:14.374244 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:14.374211    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785921c5788f789d12126ab325e03f2b22906183258a336dce129db25e5fcae1\": container with ID starting with 785921c5788f789d12126ab325e03f2b22906183258a336dce129db25e5fcae1 not found: ID does not exist" containerID="785921c5788f789d12126ab325e03f2b22906183258a336dce129db25e5fcae1"
Oct 30 00:25:14.374371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.374249    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785921c5788f789d12126ab325e03f2b22906183258a336dce129db25e5fcae1"} err="failed to get container status \"785921c5788f789d12126ab325e03f2b22906183258a336dce129db25e5fcae1\": rpc error: code = NotFound desc = could not find container \"785921c5788f789d12126ab325e03f2b22906183258a336dce129db25e5fcae1\": container with ID starting with 785921c5788f789d12126ab325e03f2b22906183258a336dce129db25e5fcae1 not found: ID does not exist"
Oct 30 00:25:14.374371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.374275    4441 scope.go:122] "RemoveContainer" containerID="4ff805a1c6fa32a453fd2739f4498fd8a9251d57189d439530ebfa5fcacc7f12"
Oct 30 00:25:14.397142 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.397099    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"]
Oct 30 00:25:14.398956 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.398926    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-cmhh5"]
Oct 30 00:25:14.443161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.424627    4441 scope.go:122] "RemoveContainer" containerID="4ff805a1c6fa32a453fd2739f4498fd8a9251d57189d439530ebfa5fcacc7f12"
Oct 30 00:25:14.443161 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:14.425116    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff805a1c6fa32a453fd2739f4498fd8a9251d57189d439530ebfa5fcacc7f12\": container with ID starting with 4ff805a1c6fa32a453fd2739f4498fd8a9251d57189d439530ebfa5fcacc7f12 not found: ID does not exist" containerID="4ff805a1c6fa32a453fd2739f4498fd8a9251d57189d439530ebfa5fcacc7f12"
Oct 30 00:25:14.443161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.425185    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff805a1c6fa32a453fd2739f4498fd8a9251d57189d439530ebfa5fcacc7f12"} err="failed to get container status \"4ff805a1c6fa32a453fd2739f4498fd8a9251d57189d439530ebfa5fcacc7f12\": rpc error: code = NotFound desc = could not find container \"4ff805a1c6fa32a453fd2739f4498fd8a9251d57189d439530ebfa5fcacc7f12\": container with ID starting with 4ff805a1c6fa32a453fd2739f4498fd8a9251d57189d439530ebfa5fcacc7f12 not found: ID does not exist"
Oct 30 00:25:14.443161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.425257    4441 scope.go:122] "RemoveContainer" containerID="f53d85cfd3fd9df994d8416d38d9143eae10b778b1ec1a62cc6ff849233c735e"
Oct 30 00:25:14.443161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.436416    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"]
Oct 30 00:25:14.448252 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.443971    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-c5h8g"]
Oct 30 00:25:14.476515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.476438    4441 scope.go:122] "RemoveContainer" containerID="f53d85cfd3fd9df994d8416d38d9143eae10b778b1ec1a62cc6ff849233c735e"
Oct 30 00:25:14.488375 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:14.488331    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53d85cfd3fd9df994d8416d38d9143eae10b778b1ec1a62cc6ff849233c735e\": container with ID starting with f53d85cfd3fd9df994d8416d38d9143eae10b778b1ec1a62cc6ff849233c735e not found: ID does not exist" containerID="f53d85cfd3fd9df994d8416d38d9143eae10b778b1ec1a62cc6ff849233c735e"
Oct 30 00:25:14.488524 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.488383    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53d85cfd3fd9df994d8416d38d9143eae10b778b1ec1a62cc6ff849233c735e"} err="failed to get container status \"f53d85cfd3fd9df994d8416d38d9143eae10b778b1ec1a62cc6ff849233c735e\": rpc error: code = NotFound desc = could not find container \"f53d85cfd3fd9df994d8416d38d9143eae10b778b1ec1a62cc6ff849233c735e\": container with ID starting with f53d85cfd3fd9df994d8416d38d9143eae10b778b1ec1a62cc6ff849233c735e not found: ID does not exist"
Oct 30 00:25:14.497151 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.497121    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"]
Oct 30 00:25:14.499322 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.499292    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-gmj7s"]
Oct 30 00:25:14.617414 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.617379    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0298952c-f699-4f6b-b32a-f35db6d0e113" path="/var/lib/kubelet/pods/0298952c-f699-4f6b-b32a-f35db6d0e113/volumes"
Oct 30 00:25:14.618805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.618776    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0360d680-dc02-4fd2-a4fd-47f584e20527" path="/var/lib/kubelet/pods/0360d680-dc02-4fd2-a4fd-47f584e20527/volumes"
Oct 30 00:25:14.619613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.619439    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dffac52a-994f-43ce-ad5b-0abf6a73f13a" path="/var/lib/kubelet/pods/dffac52a-994f-43ce-ad5b-0abf6a73f13a/volumes"
Oct 30 00:25:14.620461 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:14.620441    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9131a7c-3280-43e8-a388-e9c51a7a0389" path="/var/lib/kubelet/pods/f9131a7c-3280-43e8-a388-e9c51a7a0389/volumes"
Oct 30 00:25:18.764262 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.764228    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"]
Oct 30 00:25:18.765409 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765380    4441 cpu_manager.go:414] "containerMap: removing container" podUID="dffac52a-994f-43ce-ad5b-0abf6a73f13a" containerName="istio-proxy"
Oct 30 00:25:18.765409 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765405    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffac52a-994f-43ce-ad5b-0abf6a73f13a" containerName="istio-proxy"
Oct 30 00:25:18.765588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765424    4441 cpu_manager.go:414] "containerMap: removing container" podUID="bee7d813-eb28-46ed-b93c-f5422243a68a" containerName="maas-api"
Oct 30 00:25:18.765588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765432    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee7d813-eb28-46ed-b93c-f5422243a68a" containerName="maas-api"
Oct 30 00:25:18.765588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765468    4441 cpu_manager.go:414] "containerMap: removing container" podUID="350647e3-09d0-481a-baaf-dc0043c75ba7" containerName="payload-pre-processing"
Oct 30 00:25:18.765588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765476    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350647e3-09d0-481a-baaf-dc0043c75ba7" containerName="payload-pre-processing"
Oct 30 00:25:18.765588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765523    4441 cpu_manager.go:414] "containerMap: removing container" podUID="0298952c-f699-4f6b-b32a-f35db6d0e113" containerName="istio-proxy"
Oct 30 00:25:18.765588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765531    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0298952c-f699-4f6b-b32a-f35db6d0e113" containerName="istio-proxy"
Oct 30 00:25:18.765588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765549    4441 cpu_manager.go:414] "containerMap: removing container" podUID="0360d680-dc02-4fd2-a4fd-47f584e20527" containerName="istio-proxy"
Oct 30 00:25:18.765588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765557    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0360d680-dc02-4fd2-a4fd-47f584e20527" containerName="istio-proxy"
Oct 30 00:25:18.765588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765576    4441 cpu_manager.go:414] "containerMap: removing container" podUID="f9131a7c-3280-43e8-a388-e9c51a7a0389" containerName="maas-api"
Oct 30 00:25:18.765588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765584    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9131a7c-3280-43e8-a388-e9c51a7a0389" containerName="maas-api"
Oct 30 00:25:18.766089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765605    4441 cpu_manager.go:414] "containerMap: removing container" podUID="65f4a9c0-bdd6-45d7-b363-f64e7461530c" containerName="payload-processing"
Oct 30 00:25:18.766089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765613    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f4a9c0-bdd6-45d7-b363-f64e7461530c" containerName="payload-processing"
Oct 30 00:25:18.766089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765784    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee7d813-eb28-46ed-b93c-f5422243a68a" containerName="maas-api"
Oct 30 00:25:18.766089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765802    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="350647e3-09d0-481a-baaf-dc0043c75ba7" containerName="payload-pre-processing"
Oct 30 00:25:18.766089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765817    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0298952c-f699-4f6b-b32a-f35db6d0e113" containerName="istio-proxy"
Oct 30 00:25:18.766089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765829    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0360d680-dc02-4fd2-a4fd-47f584e20527" containerName="istio-proxy"
Oct 30 00:25:18.766089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765839    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d7723e-9a7d-4858-8093-4b13593809db" containerName="istio-proxy"
Oct 30 00:25:18.766089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765848    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9131a7c-3280-43e8-a388-e9c51a7a0389" containerName="maas-api"
Oct 30 00:25:18.766089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765860    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f4a9c0-bdd6-45d7-b363-f64e7461530c" containerName="payload-processing"
Oct 30 00:25:18.766089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.765871    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffac52a-994f-43ce-ad5b-0abf6a73f13a" containerName="istio-proxy"
Oct 30 00:25:18.771729 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.771708    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:18.802091 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.791018    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"]
Oct 30 00:25:18.802091 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.795687    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3fb3ad-serving-cert\""
Oct 30 00:25:18.866821 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.866784    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f95d5b-fbe6-4ee5-be65-54f16728355c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:18.867039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.867001    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:18.867142 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.867125    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks96f\" (UniqueName: \"kubernetes.io/projected/e2f95d5b-fbe6-4ee5-be65-54f16728355c-kube-api-access-ks96f\") pod \"maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:18.867247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.867224    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:18.968589 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.968551    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f95d5b-fbe6-4ee5-be65-54f16728355c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:18.968755 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.968705    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:18.968822 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.968776    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks96f\" (UniqueName: \"kubernetes.io/projected/e2f95d5b-fbe6-4ee5-be65-54f16728355c-kube-api-access-ks96f\") pod \"maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:18.968896 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:18.968811    4441 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-serving-cert: secret "maas-api-e2e-ipp-a-3fb3ad-serving-cert" not found
Oct 30 00:25:18.968896 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.968830    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:18.968896 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:18.968893    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-maas-api-tls podName:e2f95d5b-fbe6-4ee5-be65-54f16728355c nodeName:}" failed. No retries permitted until 2026-10-30 00:25:19.468873141 +0000 UTC m=+1228.994167011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-maas-api-tls") pod "maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb" (UID: "e2f95d5b-fbe6-4ee5-be65-54f16728355c") : secret "maas-api-e2e-ipp-a-3fb3ad-serving-cert" not found
Oct 30 00:25:18.969378 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.969353    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f95d5b-fbe6-4ee5-be65-54f16728355c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:18.971777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.971752    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:18.991634 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:18.991603    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks96f\" (UniqueName: \"kubernetes.io/projected/e2f95d5b-fbe6-4ee5-be65-54f16728355c-kube-api-access-ks96f\") pod \"maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:19.485172 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:19.485136    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:19.487709 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:19.487687    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:19.699240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:19.699208    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:20.053583 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:20.053550    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"]
Oct 30 00:25:20.054534 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:25:20.054489    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f95d5b_fbe6_4ee5_be65_54f16728355c.slice/crio-3b22ff87c2d278b5bdb8369102878fd18696029fd88228739976f25eb3bcaeb0 WatchSource:0}: Error finding container 3b22ff87c2d278b5bdb8369102878fd18696029fd88228739976f25eb3bcaeb0: Status 404 returned error can't find the container with id 3b22ff87c2d278b5bdb8369102878fd18696029fd88228739976f25eb3bcaeb0
Oct 30 00:25:20.369392 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:20.369289    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb" event={"ID":"e2f95d5b-fbe6-4ee5-be65-54f16728355c","Type":"ContainerStarted","Data":"3b22ff87c2d278b5bdb8369102878fd18696029fd88228739976f25eb3bcaeb0"}
Oct 30 00:25:21.377514 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:21.377474    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb" event={"ID":"e2f95d5b-fbe6-4ee5-be65-54f16728355c","Type":"ContainerStarted","Data":"078ce15de85c055ab6e1e889fd619a056918efb272038f3ee0f70e6335d9320c"}
Oct 30 00:25:21.377989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:21.377577    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:21.416733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:21.416683    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb" podStartSLOduration=2.935683023 podStartE2EDuration="3.416669259s" totalImagesPullingTime="480.986236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:25:21.414084241 +0000 UTC m=+1230.939378118" watchObservedRunningTime="2026-10-30 00:25:21.416669259 +0000 UTC m=+1230.941963137"
Oct 30 00:25:24.643961 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:24.643927    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"]
Oct 30 00:25:24.644674 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:24.644168    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc" podUID="b1c61570-5301-4408-bbf4-6f59bb6d4dc9" containerName="maas-api" containerID="cri-o://cd0bd65dae0f02717dd5fd1573fe25f02c64bd1636c506983f733afa0b93017d" gracePeriod=30
Oct 30 00:25:24.994316 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:24.994291    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:25:25.132047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.108818    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-metrics-certs\") pod \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") "
Oct 30 00:25:25.132047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.109015    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-maas-api-tls\") pod \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") "
Oct 30 00:25:25.132047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.109126    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-kube-api-access-gbmzk\" (UniqueName: \"kubernetes.io/projected/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-kube-api-access-gbmzk\") pod \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") "
Oct 30 00:25:25.132047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.109202    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-openshift-service-ca\") pod \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\" (UID: \"b1c61570-5301-4408-bbf4-6f59bb6d4dc9\") "
Oct 30 00:25:25.132047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.110489    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-openshift-service-ca" pod "b1c61570-5301-4408-bbf4-6f59bb6d4dc9" (UID: "b1c61570-5301-4408-bbf4-6f59bb6d4dc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:25:25.159494 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.157808    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-maas-api-tls" pod "b1c61570-5301-4408-bbf4-6f59bb6d4dc9" (UID: "b1c61570-5301-4408-bbf4-6f59bb6d4dc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:25:25.187609 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.186205    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-kube-api-access-gbmzk" pod "b1c61570-5301-4408-bbf4-6f59bb6d4dc9" (UID: "b1c61570-5301-4408-bbf4-6f59bb6d4dc9"). InnerVolumeSpecName "kube-api-access-gbmzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:25.192757 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.191185    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-metrics-certs" pod "b1c61570-5301-4408-bbf4-6f59bb6d4dc9" (UID: "b1c61570-5301-4408-bbf4-6f59bb6d4dc9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:25:25.219623 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.219526    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:25.219623 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.219567    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbmzk\" (UniqueName: \"kubernetes.io/projected/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-kube-api-access-gbmzk\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:25.219623 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.219583    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:25.219623 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.219596    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c61570-5301-4408-bbf4-6f59bb6d4dc9-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:25.410047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.410002    4441 generic.go:356] "Generic (PLEG): container finished" podID="b1c61570-5301-4408-bbf4-6f59bb6d4dc9" containerID="cd0bd65dae0f02717dd5fd1573fe25f02c64bd1636c506983f733afa0b93017d" exitCode=0
Oct 30 00:25:25.410236 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.410138    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"
Oct 30 00:25:25.410236 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.410138    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc" event={"ID":"b1c61570-5301-4408-bbf4-6f59bb6d4dc9","Type":"ContainerDied","Data":"cd0bd65dae0f02717dd5fd1573fe25f02c64bd1636c506983f733afa0b93017d"}
Oct 30 00:25:25.410394 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.410249    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc" event={"ID":"b1c61570-5301-4408-bbf4-6f59bb6d4dc9","Type":"ContainerDied","Data":"1ce0e1696ac361d2de3a0621a375ea03abb77d57c5a6f4df4b697e83580ea99e"}
Oct 30 00:25:25.410394 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.410277    4441 scope.go:122] "RemoveContainer" containerID="cd0bd65dae0f02717dd5fd1573fe25f02c64bd1636c506983f733afa0b93017d"
Oct 30 00:25:25.453384 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.453356    4441 scope.go:122] "RemoveContainer" containerID="cd0bd65dae0f02717dd5fd1573fe25f02c64bd1636c506983f733afa0b93017d"
Oct 30 00:25:25.453728 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:25.453704    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0bd65dae0f02717dd5fd1573fe25f02c64bd1636c506983f733afa0b93017d\": container with ID starting with cd0bd65dae0f02717dd5fd1573fe25f02c64bd1636c506983f733afa0b93017d not found: ID does not exist" containerID="cd0bd65dae0f02717dd5fd1573fe25f02c64bd1636c506983f733afa0b93017d"
Oct 30 00:25:25.453806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.453737    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0bd65dae0f02717dd5fd1573fe25f02c64bd1636c506983f733afa0b93017d"} err="failed to get container status \"cd0bd65dae0f02717dd5fd1573fe25f02c64bd1636c506983f733afa0b93017d\": rpc error: code = NotFound desc = could not find container \"cd0bd65dae0f02717dd5fd1573fe25f02c64bd1636c506983f733afa0b93017d\": container with ID starting with cd0bd65dae0f02717dd5fd1573fe25f02c64bd1636c506983f733afa0b93017d not found: ID does not exist"
Oct 30 00:25:25.470543 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.470462    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"]
Oct 30 00:25:25.477551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:25.477519    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f52ea1-5ffb4c4b5c-rkcsc"]
Oct 30 00:25:26.610301 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:26.610268    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c61570-5301-4408-bbf4-6f59bb6d4dc9" path="/var/lib/kubelet/pods/b1c61570-5301-4408-bbf4-6f59bb6d4dc9/volumes"
Oct 30 00:25:27.390272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:27.390244    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:25:32.765750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:32.765709    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"]
Oct 30 00:25:32.766286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:32.765972    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l" podUID="936b1610-9e63-4eb6-95c3-8d2e10d307f7" containerName="istio-proxy" containerID="cri-o://30176cca34d99e80bc6f70a5c2dbdefd480f298891c0ef54f4724549af70d15a" gracePeriod=30
Oct 30 00:25:33.944336 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:33.944290    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:25:38.072515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.072479    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:25:38.170084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.132582    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istiod-ca-cert\") pod \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") "
Oct 30 00:25:38.170084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.133099    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istiod-ca-cert" pod "936b1610-9e63-4eb6-95c3-8d2e10d307f7" (UID: "936b1610-9e63-4eb6-95c3-8d2e10d307f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:25:38.170084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.166517    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-token\" (UniqueName: \"kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-token\") pod \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") "
Oct 30 00:25:38.177326 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.177279    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-data" pod "936b1610-9e63-4eb6-95c3-8d2e10d307f7" (UID: "936b1610-9e63-4eb6-95c3-8d2e10d307f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:38.181432 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.181404    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-data\") pod \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") "
Oct 30 00:25:38.181881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.181606    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-certs\") pod \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") "
Oct 30 00:25:38.181881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.181676    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-kube-api-access-kx4l9\" (UniqueName: \"kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-kube-api-access-kx4l9\") pod \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") "
Oct 30 00:25:38.181881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.181717    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-socket\") pod \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") "
Oct 30 00:25:38.181881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.181783    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-podinfo\") pod \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") "
Oct 30 00:25:38.182465 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.182430    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-certs" pod "936b1610-9e63-4eb6-95c3-8d2e10d307f7" (UID: "936b1610-9e63-4eb6-95c3-8d2e10d307f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:38.182990 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.182964    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-socket" pod "936b1610-9e63-4eb6-95c3-8d2e10d307f7" (UID: "936b1610-9e63-4eb6-95c3-8d2e10d307f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:38.183324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.183301    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-credential-socket\") pod \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") "
Oct 30 00:25:38.183451 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.183433    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-envoy\") pod \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\" (UID: \"936b1610-9e63-4eb6-95c3-8d2e10d307f7\") "
Oct 30 00:25:38.188426 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.184639    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-credential-socket" pod "936b1610-9e63-4eb6-95c3-8d2e10d307f7" (UID: "936b1610-9e63-4eb6-95c3-8d2e10d307f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:38.188426 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.184979    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:38.188426 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.185000    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:38.188426 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.185014    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:38.188426 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.185051    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:38.188426 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.185064    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:38.190635 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.190606    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-token" pod "936b1610-9e63-4eb6-95c3-8d2e10d307f7" (UID: "936b1610-9e63-4eb6-95c3-8d2e10d307f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:38.190741 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.190632    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-kube-api-access-kx4l9" pod "936b1610-9e63-4eb6-95c3-8d2e10d307f7" (UID: "936b1610-9e63-4eb6-95c3-8d2e10d307f7"). InnerVolumeSpecName "kube-api-access-kx4l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:38.191123 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.191095    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-podinfo" pod "936b1610-9e63-4eb6-95c3-8d2e10d307f7" (UID: "936b1610-9e63-4eb6-95c3-8d2e10d307f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:25:38.192476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.192456    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-envoy" pod "936b1610-9e63-4eb6-95c3-8d2e10d307f7" (UID: "936b1610-9e63-4eb6-95c3-8d2e10d307f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:38.286887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.286804    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx4l9\" (UniqueName: \"kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-kube-api-access-kx4l9\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:38.286887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.286835    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:38.286887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.286845    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:38.286887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.286853    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936b1610-9e63-4eb6-95c3-8d2e10d307f7-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:38.569317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.569230    4441 generic.go:356] "Generic (PLEG): container finished" podID="936b1610-9e63-4eb6-95c3-8d2e10d307f7" containerID="30176cca34d99e80bc6f70a5c2dbdefd480f298891c0ef54f4724549af70d15a" exitCode=0
Oct 30 00:25:38.569317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.569293    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"
Oct 30 00:25:38.569558 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.569316    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l" event={"ID":"936b1610-9e63-4eb6-95c3-8d2e10d307f7","Type":"ContainerDied","Data":"30176cca34d99e80bc6f70a5c2dbdefd480f298891c0ef54f4724549af70d15a"}
Oct 30 00:25:38.569558 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.569384    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l" event={"ID":"936b1610-9e63-4eb6-95c3-8d2e10d307f7","Type":"ContainerDied","Data":"79eabcad0030ed7ab971c9b28673d2556f6e901ba020bbd131d204dfc84369a0"}
Oct 30 00:25:38.569558 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.569414    4441 scope.go:122] "RemoveContainer" containerID="30176cca34d99e80bc6f70a5c2dbdefd480f298891c0ef54f4724549af70d15a"
Oct 30 00:25:38.581425 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.581409    4441 scope.go:122] "RemoveContainer" containerID="30176cca34d99e80bc6f70a5c2dbdefd480f298891c0ef54f4724549af70d15a"
Oct 30 00:25:38.581722 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:38.581699    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30176cca34d99e80bc6f70a5c2dbdefd480f298891c0ef54f4724549af70d15a\": container with ID starting with 30176cca34d99e80bc6f70a5c2dbdefd480f298891c0ef54f4724549af70d15a not found: ID does not exist" containerID="30176cca34d99e80bc6f70a5c2dbdefd480f298891c0ef54f4724549af70d15a"
Oct 30 00:25:38.581781 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.581729    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30176cca34d99e80bc6f70a5c2dbdefd480f298891c0ef54f4724549af70d15a"} err="failed to get container status \"30176cca34d99e80bc6f70a5c2dbdefd480f298891c0ef54f4724549af70d15a\": rpc error: code = NotFound desc = could not find container \"30176cca34d99e80bc6f70a5c2dbdefd480f298891c0ef54f4724549af70d15a\": container with ID starting with 30176cca34d99e80bc6f70a5c2dbdefd480f298891c0ef54f4724549af70d15a not found: ID does not exist"
Oct 30 00:25:38.629249 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.629218    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"]
Oct 30 00:25:38.632872 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:38.632848    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-nkl2l"]
Oct 30 00:25:40.611156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:40.611126    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936b1610-9e63-4eb6-95c3-8d2e10d307f7" path="/var/lib/kubelet/pods/936b1610-9e63-4eb6-95c3-8d2e10d307f7/volumes"
Oct 30 00:25:50.140703 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:50.140670    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:25:50.328049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:50.327309    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"]
Oct 30 00:25:50.328049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:50.327558    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf" podUID="c81d80c1-da80-4408-8cdf-4bd4acbbfc84" containerName="istio-proxy" containerID="cri-o://3ae796120c195ec36f3322c2297d24acd5399dceb59d2745ebcda693793ffad3" gracePeriod=30
Oct 30 00:25:50.334867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:50.332393    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"]
Oct 30 00:25:50.334867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:50.332720    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" containerName="istio-proxy" containerID="cri-o://273d59d172fd6f5d2cce2e65d7a1ecc0da601c15e29742629b6399f6143cc1d5" gracePeriod=30
Oct 30 00:25:50.341956 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:50.337215    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"]
Oct 30 00:25:50.341956 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:50.337400    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy" containerID="cri-o://c4bc658687983639c999c91890cfdf9bda947381fde4f2ea87fb9230aa440ab6" gracePeriod=30
Oct 30 00:25:51.114096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:51.114053    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:25:51.114278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:51.114125    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:25:51.856329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:51.856290    4441 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:25:51.856831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:51.856354    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:25:55.713093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.713062    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:25:55.720605 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.720573    4441 generic.go:356] "Generic (PLEG): container finished" podID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerID="c4bc658687983639c999c91890cfdf9bda947381fde4f2ea87fb9230aa440ab6" exitCode=0
Oct 30 00:25:55.720729 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.720644    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" event={"ID":"d6544480-6c61-4c34-a724-dc84f756a9e6","Type":"ContainerDied","Data":"c4bc658687983639c999c91890cfdf9bda947381fde4f2ea87fb9230aa440ab6"}
Oct 30 00:25:55.728739 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.728713    4441 generic.go:356] "Generic (PLEG): container finished" podID="c81d80c1-da80-4408-8cdf-4bd4acbbfc84" containerID="3ae796120c195ec36f3322c2297d24acd5399dceb59d2745ebcda693793ffad3" exitCode=0
Oct 30 00:25:55.728875 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.728815    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf" event={"ID":"c81d80c1-da80-4408-8cdf-4bd4acbbfc84","Type":"ContainerDied","Data":"3ae796120c195ec36f3322c2297d24acd5399dceb59d2745ebcda693793ffad3"}
Oct 30 00:25:55.728875 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.728861    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf" event={"ID":"c81d80c1-da80-4408-8cdf-4bd4acbbfc84","Type":"ContainerDied","Data":"11852715374fb701d40e57d5127c3ab284fb14628178549b87c1ccd826c63a9c"}
Oct 30 00:25:55.729075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.728891    4441 scope.go:122] "RemoveContainer" containerID="3ae796120c195ec36f3322c2297d24acd5399dceb59d2745ebcda693793ffad3"
Oct 30 00:25:55.729075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.729067    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"
Oct 30 00:25:55.734857 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.734833    4441 generic.go:356] "Generic (PLEG): container finished" podID="27f48a0a-3901-46b0-adc0-4f447a436671" containerID="273d59d172fd6f5d2cce2e65d7a1ecc0da601c15e29742629b6399f6143cc1d5" exitCode=0
Oct 30 00:25:55.735078 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.735015    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" event={"ID":"27f48a0a-3901-46b0-adc0-4f447a436671","Type":"ContainerDied","Data":"273d59d172fd6f5d2cce2e65d7a1ecc0da601c15e29742629b6399f6143cc1d5"}
Oct 30 00:25:55.739063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.739043    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:25:55.762184 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.762159    4441 scope.go:122] "RemoveContainer" containerID="3ae796120c195ec36f3322c2297d24acd5399dceb59d2745ebcda693793ffad3"
Oct 30 00:25:55.777227 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:55.777199    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae796120c195ec36f3322c2297d24acd5399dceb59d2745ebcda693793ffad3\": container with ID starting with 3ae796120c195ec36f3322c2297d24acd5399dceb59d2745ebcda693793ffad3 not found: ID does not exist" containerID="3ae796120c195ec36f3322c2297d24acd5399dceb59d2745ebcda693793ffad3"
Oct 30 00:25:55.777351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.777235    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae796120c195ec36f3322c2297d24acd5399dceb59d2745ebcda693793ffad3"} err="failed to get container status \"3ae796120c195ec36f3322c2297d24acd5399dceb59d2745ebcda693793ffad3\": rpc error: code = NotFound desc = could not find container \"3ae796120c195ec36f3322c2297d24acd5399dceb59d2745ebcda693793ffad3\": container with ID starting with 3ae796120c195ec36f3322c2297d24acd5399dceb59d2745ebcda693793ffad3 not found: ID does not exist"
Oct 30 00:25:55.806865 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.806841    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847175    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-socket\") pod \"27f48a0a-3901-46b0-adc0-4f447a436671\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847288    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-kube-api-access-z5pzt\" (UniqueName: \"kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-kube-api-access-z5pzt\") pod \"27f48a0a-3901-46b0-adc0-4f447a436671\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847339    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-envoy\") pod \"27f48a0a-3901-46b0-adc0-4f447a436671\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847382    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-envoy\") pod \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847439    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-certs\") pod \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847452    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-socket" pod "27f48a0a-3901-46b0-adc0-4f447a436671" (UID: "27f48a0a-3901-46b0-adc0-4f447a436671"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847508    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f48a0a-3901-46b0-adc0-4f447a436671-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f48a0a-3901-46b0-adc0-4f447a436671-istiod-ca-cert\") pod \"27f48a0a-3901-46b0-adc0-4f447a436671\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847585    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27f48a0a-3901-46b0-adc0-4f447a436671-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f48a0a-3901-46b0-adc0-4f447a436671-istio-podinfo\") pod \"27f48a0a-3901-46b0-adc0-4f447a436671\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847651    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-socket\") pod \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847689    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-data\") pod \"27f48a0a-3901-46b0-adc0-4f447a436671\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847745    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-istio-token\" (UniqueName: \"kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-istio-token\") pod \"27f48a0a-3901-46b0-adc0-4f447a436671\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847817    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istiod-ca-cert\") pod \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847865    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-token\" (UniqueName: \"kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-token\") pod \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847894    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-credential-socket\") pod \"27f48a0a-3901-46b0-adc0-4f447a436671\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.847979    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-kube-api-access-w48vf\" (UniqueName: \"kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-kube-api-access-w48vf\") pod \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") "
Oct 30 00:25:55.848006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.848007    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-data\") pod \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") "
Oct 30 00:25:55.848993 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.848205    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f48a0a-3901-46b0-adc0-4f447a436671-istiod-ca-cert" pod "27f48a0a-3901-46b0-adc0-4f447a436671" (UID: "27f48a0a-3901-46b0-adc0-4f447a436671"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:25:55.848993 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.848511    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-certs" pod "c81d80c1-da80-4408-8cdf-4bd4acbbfc84" (UID: "c81d80c1-da80-4408-8cdf-4bd4acbbfc84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.848993 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.848768    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-socket" pod "c81d80c1-da80-4408-8cdf-4bd4acbbfc84" (UID: "c81d80c1-da80-4408-8cdf-4bd4acbbfc84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.848993 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.848933    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-certs\") pod \"27f48a0a-3901-46b0-adc0-4f447a436671\" (UID: \"27f48a0a-3901-46b0-adc0-4f447a436671\") "
Oct 30 00:25:55.849235 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.849006    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-podinfo\") pod \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") "
Oct 30 00:25:55.849235 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.849058    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-credential-socket\") pod \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\" (UID: \"c81d80c1-da80-4408-8cdf-4bd4acbbfc84\") "
Oct 30 00:25:55.849235 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.849191    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-credential-socket" pod "27f48a0a-3901-46b0-adc0-4f447a436671" (UID: "27f48a0a-3901-46b0-adc0-4f447a436671"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.849712 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.849477    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istiod-ca-cert" pod "c81d80c1-da80-4408-8cdf-4bd4acbbfc84" (UID: "c81d80c1-da80-4408-8cdf-4bd4acbbfc84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:25:55.849712 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.849692    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-credential-socket" pod "c81d80c1-da80-4408-8cdf-4bd4acbbfc84" (UID: "c81d80c1-da80-4408-8cdf-4bd4acbbfc84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.850515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.850147    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.850515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.850175    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.850515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.850190    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.850515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.850205    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.850515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.850219    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.850515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.850231    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.850515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.850244    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f48a0a-3901-46b0-adc0-4f447a436671-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.851164 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.851141    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-certs" pod "27f48a0a-3901-46b0-adc0-4f447a436671" (UID: "27f48a0a-3901-46b0-adc0-4f447a436671"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.851461 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.851444    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-data" pod "c81d80c1-da80-4408-8cdf-4bd4acbbfc84" (UID: "c81d80c1-da80-4408-8cdf-4bd4acbbfc84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.851764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.851744    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-data" pod "27f48a0a-3901-46b0-adc0-4f447a436671" (UID: "27f48a0a-3901-46b0-adc0-4f447a436671"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.857482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.857362    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-envoy" pod "27f48a0a-3901-46b0-adc0-4f447a436671" (UID: "27f48a0a-3901-46b0-adc0-4f447a436671"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.857482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.857430    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-token" pod "c81d80c1-da80-4408-8cdf-4bd4acbbfc84" (UID: "c81d80c1-da80-4408-8cdf-4bd4acbbfc84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:55.857482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.857449    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27f48a0a-3901-46b0-adc0-4f447a436671-istio-podinfo" pod "27f48a0a-3901-46b0-adc0-4f447a436671" (UID: "27f48a0a-3901-46b0-adc0-4f447a436671"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:25:55.857482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.857459    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-kube-api-access-w48vf" pod "c81d80c1-da80-4408-8cdf-4bd4acbbfc84" (UID: "c81d80c1-da80-4408-8cdf-4bd4acbbfc84"). InnerVolumeSpecName "kube-api-access-w48vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:55.857734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.857554    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-istio-token" pod "27f48a0a-3901-46b0-adc0-4f447a436671" (UID: "27f48a0a-3901-46b0-adc0-4f447a436671"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:55.857734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.857580    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-kube-api-access-z5pzt" pod "27f48a0a-3901-46b0-adc0-4f447a436671" (UID: "27f48a0a-3901-46b0-adc0-4f447a436671"). InnerVolumeSpecName "kube-api-access-z5pzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:55.857734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.857700    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-envoy" pod "c81d80c1-da80-4408-8cdf-4bd4acbbfc84" (UID: "c81d80c1-da80-4408-8cdf-4bd4acbbfc84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.861013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.860991    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-podinfo" pod "c81d80c1-da80-4408-8cdf-4bd4acbbfc84" (UID: "c81d80c1-da80-4408-8cdf-4bd4acbbfc84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:25:55.950846 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.950763    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-credential-socket\") pod \"d6544480-6c61-4c34-a724-dc84f756a9e6\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") "
Oct 30 00:25:55.950846 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.950811    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6544480-6c61-4c34-a724-dc84f756a9e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6544480-6c61-4c34-a724-dc84f756a9e6-istiod-ca-cert\") pod \"d6544480-6c61-4c34-a724-dc84f756a9e6\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") "
Oct 30 00:25:55.951079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.950907    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-token\" (UniqueName: \"kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-token\") pod \"d6544480-6c61-4c34-a724-dc84f756a9e6\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") "
Oct 30 00:25:55.951079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.950991    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-certs\") pod \"d6544480-6c61-4c34-a724-dc84f756a9e6\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") "
Oct 30 00:25:55.951198 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.951083    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-envoy\") pod \"d6544480-6c61-4c34-a724-dc84f756a9e6\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") "
Oct 30 00:25:55.951198 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.951108    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-credential-socket" pod "d6544480-6c61-4c34-a724-dc84f756a9e6" (UID: "d6544480-6c61-4c34-a724-dc84f756a9e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.951198 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.951147    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-kube-api-access-w7g7r\" (UniqueName: \"kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-kube-api-access-w7g7r\") pod \"d6544480-6c61-4c34-a724-dc84f756a9e6\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") "
Oct 30 00:25:55.951198 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.951181    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-socket\") pod \"d6544480-6c61-4c34-a724-dc84f756a9e6\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") "
Oct 30 00:25:55.951419 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.951209    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-podinfo\") pod \"d6544480-6c61-4c34-a724-dc84f756a9e6\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") "
Oct 30 00:25:55.951419 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.951254    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-certs" pod "d6544480-6c61-4c34-a724-dc84f756a9e6" (UID: "d6544480-6c61-4c34-a724-dc84f756a9e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.951419 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.951260    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-data\") pod \"d6544480-6c61-4c34-a724-dc84f756a9e6\" (UID: \"d6544480-6c61-4c34-a724-dc84f756a9e6\") "
Oct 30 00:25:55.951419 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.951300    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6544480-6c61-4c34-a724-dc84f756a9e6-istiod-ca-cert" pod "d6544480-6c61-4c34-a724-dc84f756a9e6" (UID: "d6544480-6c61-4c34-a724-dc84f756a9e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:25:55.951634 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.951558    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-socket" pod "d6544480-6c61-4c34-a724-dc84f756a9e6" (UID: "d6544480-6c61-4c34-a724-dc84f756a9e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.951942 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.951919    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-data" pod "d6544480-6c61-4c34-a724-dc84f756a9e6" (UID: "d6544480-6c61-4c34-a724-dc84f756a9e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.952261 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952240    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5pzt\" (UniqueName: \"kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-kube-api-access-z5pzt\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952268    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952283    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952297    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952309    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6544480-6c61-4c34-a724-dc84f756a9e6-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952338    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f48a0a-3901-46b0-adc0-4f447a436671-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952350    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952371    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952383    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f48a0a-3901-46b0-adc0-4f447a436671-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952396    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952409    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952421    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952434    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w48vf\" (UniqueName: \"kubernetes.io/projected/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-kube-api-access-w48vf\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952448    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952463    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f48a0a-3901-46b0-adc0-4f447a436671-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.952607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.952475    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81d80c1-da80-4408-8cdf-4bd4acbbfc84-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:55.953312 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.953285    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-token" pod "d6544480-6c61-4c34-a724-dc84f756a9e6" (UID: "d6544480-6c61-4c34-a724-dc84f756a9e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:55.953488 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.953464    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-envoy" pod "d6544480-6c61-4c34-a724-dc84f756a9e6" (UID: "d6544480-6c61-4c34-a724-dc84f756a9e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:25:55.953863 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.953842    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-podinfo" pod "d6544480-6c61-4c34-a724-dc84f756a9e6" (UID: "d6544480-6c61-4c34-a724-dc84f756a9e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:25:55.954221 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:55.954196    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-kube-api-access-w7g7r" pod "d6544480-6c61-4c34-a724-dc84f756a9e6" (UID: "d6544480-6c61-4c34-a724-dc84f756a9e6"). InnerVolumeSpecName "kube-api-access-w7g7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:56.054398 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.054357    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:56.054398 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.054394    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:56.054398 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.054409    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7g7r\" (UniqueName: \"kubernetes.io/projected/d6544480-6c61-4c34-a724-dc84f756a9e6-kube-api-access-w7g7r\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:56.054585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.054420    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6544480-6c61-4c34-a724-dc84f756a9e6-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:56.075228 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.075201    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"]
Oct 30 00:25:56.078648 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.078625    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-r5nhf"]
Oct 30 00:25:56.611099 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.611068    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81d80c1-da80-4408-8cdf-4bd4acbbfc84" path="/var/lib/kubelet/pods/c81d80c1-da80-4408-8cdf-4bd4acbbfc84/volumes"
Oct 30 00:25:56.745783 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.745757    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"
Oct 30 00:25:56.746277 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.745783    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf" event={"ID":"27f48a0a-3901-46b0-adc0-4f447a436671","Type":"ContainerDied","Data":"aa9757de0bb19442fac539d98e441673c53e366c5127d5a7a7aa823a4ca16afb"}
Oct 30 00:25:56.746277 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.745851    4441 scope.go:122] "RemoveContainer" containerID="273d59d172fd6f5d2cce2e65d7a1ecc0da601c15e29742629b6399f6143cc1d5"
Oct 30 00:25:56.747614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.747581    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm" event={"ID":"d6544480-6c61-4c34-a724-dc84f756a9e6","Type":"ContainerDied","Data":"1508ff44b3c62c9abc1581e937283e4233cf89ffe86d31c1bf2d07721608a5e3"}
Oct 30 00:25:56.747614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.747605    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"
Oct 30 00:25:56.759201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.759180    4441 scope.go:122] "RemoveContainer" containerID="c4bc658687983639c999c91890cfdf9bda947381fde4f2ea87fb9230aa440ab6"
Oct 30 00:25:56.796316 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.796285    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"]
Oct 30 00:25:56.810340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.810306    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-ptsxm"]
Oct 30 00:25:56.835815 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.835782    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"]
Oct 30 00:25:56.840703 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:56.840677    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-98kmf"]
Oct 30 00:25:57.970897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:57.970866    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"]
Oct 30 00:25:57.971524 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:57.971083    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8" podUID="02834ba0-2a7a-4e99-8d35-d9b15c9cef67" containerName="maas-api" containerID="cri-o://f570617066c57f7b011066c97c15385c3de77da3c7e8cb1d6d9d560e802e37fe" gracePeriod=30
Oct 30 00:25:58.324322 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.324298    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:25:58.490014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.489980    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-metrics-certs\") pod \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") "
Oct 30 00:25:58.490227 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.490156    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-kube-api-access-7zjfv\" (UniqueName: \"kubernetes.io/projected/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-kube-api-access-7zjfv\") pod \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") "
Oct 30 00:25:58.490227 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.490181    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-openshift-service-ca\") pod \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") "
Oct 30 00:25:58.490319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.490237    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-maas-api-tls\") pod \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\" (UID: \"02834ba0-2a7a-4e99-8d35-d9b15c9cef67\") "
Oct 30 00:25:58.490607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.490582    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-openshift-service-ca" pod "02834ba0-2a7a-4e99-8d35-d9b15c9cef67" (UID: "02834ba0-2a7a-4e99-8d35-d9b15c9cef67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:25:58.490936 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.490914    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:58.492304 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.492276    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-metrics-certs" pod "02834ba0-2a7a-4e99-8d35-d9b15c9cef67" (UID: "02834ba0-2a7a-4e99-8d35-d9b15c9cef67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:25:58.492416 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.492352    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-kube-api-access-7zjfv" pod "02834ba0-2a7a-4e99-8d35-d9b15c9cef67" (UID: "02834ba0-2a7a-4e99-8d35-d9b15c9cef67"). InnerVolumeSpecName "kube-api-access-7zjfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:25:58.492475 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.492458    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-maas-api-tls" pod "02834ba0-2a7a-4e99-8d35-d9b15c9cef67" (UID: "02834ba0-2a7a-4e99-8d35-d9b15c9cef67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:25:58.592319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.592246    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:58.592319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.592276    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zjfv\" (UniqueName: \"kubernetes.io/projected/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-kube-api-access-7zjfv\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:58.592319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.592287    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02834ba0-2a7a-4e99-8d35-d9b15c9cef67-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:25:58.611991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.611953    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" path="/var/lib/kubelet/pods/27f48a0a-3901-46b0-adc0-4f447a436671/volumes"
Oct 30 00:25:58.612697 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.612672    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" path="/var/lib/kubelet/pods/d6544480-6c61-4c34-a724-dc84f756a9e6/volumes"
Oct 30 00:25:58.772698 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.772661    4441 generic.go:356] "Generic (PLEG): container finished" podID="02834ba0-2a7a-4e99-8d35-d9b15c9cef67" containerID="f570617066c57f7b011066c97c15385c3de77da3c7e8cb1d6d9d560e802e37fe" exitCode=0
Oct 30 00:25:58.772903 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.772884    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8" event={"ID":"02834ba0-2a7a-4e99-8d35-d9b15c9cef67","Type":"ContainerDied","Data":"f570617066c57f7b011066c97c15385c3de77da3c7e8cb1d6d9d560e802e37fe"}
Oct 30 00:25:58.772977 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.772939    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8" event={"ID":"02834ba0-2a7a-4e99-8d35-d9b15c9cef67","Type":"ContainerDied","Data":"c296ce844115849d48c163f7d68246af8ac8f2e578d5cdd4ab473ef6ab36f2e9"}
Oct 30 00:25:58.772977 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.772969    4441 scope.go:122] "RemoveContainer" containerID="f570617066c57f7b011066c97c15385c3de77da3c7e8cb1d6d9d560e802e37fe"
Oct 30 00:25:58.773167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.773151    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"
Oct 30 00:25:58.789572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.789520    4441 scope.go:122] "RemoveContainer" containerID="f570617066c57f7b011066c97c15385c3de77da3c7e8cb1d6d9d560e802e37fe"
Oct 30 00:25:58.789993 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:25:58.789961    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f570617066c57f7b011066c97c15385c3de77da3c7e8cb1d6d9d560e802e37fe\": container with ID starting with f570617066c57f7b011066c97c15385c3de77da3c7e8cb1d6d9d560e802e37fe not found: ID does not exist" containerID="f570617066c57f7b011066c97c15385c3de77da3c7e8cb1d6d9d560e802e37fe"
Oct 30 00:25:58.790131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.790001    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f570617066c57f7b011066c97c15385c3de77da3c7e8cb1d6d9d560e802e37fe"} err="failed to get container status \"f570617066c57f7b011066c97c15385c3de77da3c7e8cb1d6d9d560e802e37fe\": rpc error: code = NotFound desc = could not find container \"f570617066c57f7b011066c97c15385c3de77da3c7e8cb1d6d9d560e802e37fe\": container with ID starting with f570617066c57f7b011066c97c15385c3de77da3c7e8cb1d6d9d560e802e37fe not found: ID does not exist"
Oct 30 00:25:58.819800 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.819770    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"]
Oct 30 00:25:58.831071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:58.825871    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a55e1-59c5b5c8b5-9s4p8"]
Oct 30 00:25:59.364927 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.364887    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"]
Oct 30 00:25:59.365841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.365817    4441 cpu_manager.go:414] "containerMap: removing container" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" containerName="istio-proxy"
Oct 30 00:25:59.365940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.365843    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" containerName="istio-proxy"
Oct 30 00:25:59.365940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.365865    4441 cpu_manager.go:414] "containerMap: removing container" podUID="c81d80c1-da80-4408-8cdf-4bd4acbbfc84" containerName="istio-proxy"
Oct 30 00:25:59.365940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.365873    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81d80c1-da80-4408-8cdf-4bd4acbbfc84" containerName="istio-proxy"
Oct 30 00:25:59.365940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.365884    4441 cpu_manager.go:414] "containerMap: removing container" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy"
Oct 30 00:25:59.365940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.365892    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy"
Oct 30 00:25:59.365940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.365901    4441 cpu_manager.go:414] "containerMap: removing container" podUID="b1c61570-5301-4408-bbf4-6f59bb6d4dc9" containerName="maas-api"
Oct 30 00:25:59.365940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.365910    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c61570-5301-4408-bbf4-6f59bb6d4dc9" containerName="maas-api"
Oct 30 00:25:59.366339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.365958    4441 cpu_manager.go:414] "containerMap: removing container" podUID="02834ba0-2a7a-4e99-8d35-d9b15c9cef67" containerName="maas-api"
Oct 30 00:25:59.366339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.365967    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02834ba0-2a7a-4e99-8d35-d9b15c9cef67" containerName="maas-api"
Oct 30 00:25:59.366339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.365985    4441 cpu_manager.go:414] "containerMap: removing container" podUID="936b1610-9e63-4eb6-95c3-8d2e10d307f7" containerName="istio-proxy"
Oct 30 00:25:59.366339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.365992    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936b1610-9e63-4eb6-95c3-8d2e10d307f7" containerName="istio-proxy"
Oct 30 00:25:59.366339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.366179    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f48a0a-3901-46b0-adc0-4f447a436671" containerName="istio-proxy"
Oct 30 00:25:59.366339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.366196    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81d80c1-da80-4408-8cdf-4bd4acbbfc84" containerName="istio-proxy"
Oct 30 00:25:59.366339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.366205    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6544480-6c61-4c34-a724-dc84f756a9e6" containerName="istio-proxy"
Oct 30 00:25:59.366339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.366233    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c61570-5301-4408-bbf4-6f59bb6d4dc9" containerName="maas-api"
Oct 30 00:25:59.366339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.366248    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="02834ba0-2a7a-4e99-8d35-d9b15c9cef67" containerName="maas-api"
Oct 30 00:25:59.366339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.366260    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="936b1610-9e63-4eb6-95c3-8d2e10d307f7" containerName="istio-proxy"
Oct 30 00:25:59.372259 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.372234    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.392403 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.391957    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-bb16a8-openshift-default-dockercfg-mp4wm\""
Oct 30 00:25:59.410355 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.410313    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"]
Oct 30 00:25:59.516975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.511322    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-data\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.516975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.511479    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-envoy\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.516975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.511527    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-token\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.516975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.511606    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqlg\" (UniqueName: \"kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-kube-api-access-xnqlg\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.516975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.511664    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-socket\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.516975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.511727    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-certs\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.516975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.511803    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istiod-ca-cert\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.516975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.511834    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-credential-socket\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.516975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.511917    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-podinfo\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.613251 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.613218    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-data\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.613465 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.613310    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-envoy\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.613465 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.613346    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-token\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.613465 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.613404    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqlg\" (UniqueName: \"kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-kube-api-access-xnqlg\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.613465 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.613453    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-socket\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.613695 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.613505    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-certs\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.613695 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.613570    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istiod-ca-cert\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.613695 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.613601    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-credential-socket\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.613695 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.613659    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-data\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.613695 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.613679    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-podinfo\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.613971 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.613944    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-socket\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.614264 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.614233    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-credential-socket\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.614264 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.614252    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-certs\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.614547 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.614519    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istiod-ca-cert\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.615837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.615768    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-envoy\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.616373 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.616343    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-podinfo\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.643797 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.643766    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqlg\" (UniqueName: \"kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-kube-api-access-xnqlg\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.646071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.644779    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-token\") pod \"e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") " pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:25:59.738349 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:25:59.738314    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:26:00.085015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:00.084984    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"]
Oct 30 00:26:00.086669 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:26:00.086641    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a972f67_aa0b_4444_9b0a_9662f67e6cac.slice/crio-3ba47d656806816d102dd2e2383c16921def39984bc92b70b933f086f3f22770 WatchSource:0}: Error finding container 3ba47d656806816d102dd2e2383c16921def39984bc92b70b933f086f3f22770: Status 404 returned error can't find the container with id 3ba47d656806816d102dd2e2383c16921def39984bc92b70b933f086f3f22770
Oct 30 00:26:00.089729 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:00.089695    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:26:00.089831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:00.089773    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:26:00.089831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:00.089811    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:26:00.610680 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:00.610648    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02834ba0-2a7a-4e99-8d35-d9b15c9cef67" path="/var/lib/kubelet/pods/02834ba0-2a7a-4e99-8d35-d9b15c9cef67/volumes"
Oct 30 00:26:00.795079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:00.795046    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" event={"ID":"4a972f67-aa0b-4444-9b0a-9662f67e6cac","Type":"ContainerStarted","Data":"58987af5c64e3fa80dd4753614ce45bfded62887cdb075f2e662b7b95bf94211"}
Oct 30 00:26:00.795248 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:00.795101    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" event={"ID":"4a972f67-aa0b-4444-9b0a-9662f67e6cac","Type":"ContainerStarted","Data":"3ba47d656806816d102dd2e2383c16921def39984bc92b70b933f086f3f22770"}
Oct 30 00:26:00.838004 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:00.837956    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" podStartSLOduration=1.83794265 podStartE2EDuration="1.83794265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:00.834336679 +0000 UTC m=+1270.359630556" watchObservedRunningTime="2026-10-30 00:26:00.83794265 +0000 UTC m=+1270.363236526"
Oct 30 00:26:01.738641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:01.738613    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:26:01.745618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:01.745592    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:26:01.808327 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:01.808267    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:26:01.809498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:01.809463    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:26:02.454087 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:02.442738    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"]
Oct 30 00:26:02.454087 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:02.448984    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"
Oct 30 00:26:02.477567 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:02.471969    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f8397b\""
Oct 30 00:26:02.477567 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:02.472089    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f8397b-dockercfg-xh924\""
Oct 30 00:26:02.477567 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:02.477280    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"]
Oct 30 00:26:02.498698 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:02.498661    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcvrw\" (UniqueName: \"kubernetes.io/projected/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-kube-api-access-hcvrw\") pod \"payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2\" (UID: \"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"
Oct 30 00:26:02.498880 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:02.498755    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2\" (UID: \"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"
Oct 30 00:26:02.604761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:02.604719    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcvrw\" (UniqueName: \"kubernetes.io/projected/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-kube-api-access-hcvrw\") pod \"payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2\" (UID: \"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"
Oct 30 00:26:02.605402 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:02.605376    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2\" (UID: \"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"
Oct 30 00:26:02.607740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:02.606889    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2\" (UID: \"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"
Oct 30 00:26:02.627379 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:02.625662    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcvrw\" (UniqueName: \"kubernetes.io/projected/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-kube-api-access-hcvrw\") pod \"payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2\" (UID: \"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"
Oct 30 00:26:02.815526 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:02.815496    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"
Oct 30 00:26:03.143437 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:03.143363    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"]
Oct 30 00:26:03.144733 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:26:03.144701    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc554ae97_8f0b_4bf9_bf77_6ac3bfb42575.slice/crio-48b95683db44c8f6d5f071979890a597f68d791bc6267248f178eb7e75ccace0 WatchSource:0}: Error finding container 48b95683db44c8f6d5f071979890a597f68d791bc6267248f178eb7e75ccace0: Status 404 returned error can't find the container with id 48b95683db44c8f6d5f071979890a597f68d791bc6267248f178eb7e75ccace0
Oct 30 00:26:03.147114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:03.147094    4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:26:03.830218 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:03.830177    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2" event={"ID":"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575","Type":"ContainerStarted","Data":"48b95683db44c8f6d5f071979890a597f68d791bc6267248f178eb7e75ccace0"}
Oct 30 00:26:03.909083 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:03.907817    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"]
Oct 30 00:26:03.909083 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:03.908111    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy" containerID="cri-o://e742d6d453a430097636f549e1de20335941b668c586f7d071b1969e85981e89" gracePeriod=30
Oct 30 00:26:03.966574 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:26:03.966369    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:26:05.847610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:05.847562    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2" event={"ID":"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575","Type":"ContainerStarted","Data":"e6f7b78f32afa089089f3b5d7f9ab99c9e81cf084bcf926d7df5881503abf659"}
Oct 30 00:26:05.849978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:05.849954    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"
Oct 30 00:26:08.030940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.030833    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2" podStartSLOduration=3.73298559 podStartE2EDuration="6.030813178s" totalImagesPullingTime="2.297827588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:05.893921138 +0000 UTC m=+1275.419215017" watchObservedRunningTime="2026-10-30 00:26:08.030813178 +0000 UTC m=+1277.556107055"
Oct 30 00:26:08.032517 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.032486    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"]
Oct 30 00:26:08.037116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.037097    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"
Oct 30 00:26:08.060966 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.060936    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fb3ad-dockercfg-5jpbd\""
Oct 30 00:26:08.061072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.060969    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fb3ad\""
Oct 30 00:26:08.067170 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.067139    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"]
Oct 30 00:26:08.139525 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.139479    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8480250-bbbd-44f3-a690-b5d2cb97efda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw\" (UID: \"e8480250-bbbd-44f3-a690-b5d2cb97efda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"
Oct 30 00:26:08.139846 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.139817    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqfcj\" (UniqueName: \"kubernetes.io/projected/e8480250-bbbd-44f3-a690-b5d2cb97efda-kube-api-access-cqfcj\") pod \"payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw\" (UID: \"e8480250-bbbd-44f3-a690-b5d2cb97efda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"
Oct 30 00:26:08.241715 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.241668    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqfcj\" (UniqueName: \"kubernetes.io/projected/e8480250-bbbd-44f3-a690-b5d2cb97efda-kube-api-access-cqfcj\") pod \"payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw\" (UID: \"e8480250-bbbd-44f3-a690-b5d2cb97efda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"
Oct 30 00:26:08.241905 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.241793    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8480250-bbbd-44f3-a690-b5d2cb97efda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw\" (UID: \"e8480250-bbbd-44f3-a690-b5d2cb97efda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"
Oct 30 00:26:08.242558 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.242534    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8480250-bbbd-44f3-a690-b5d2cb97efda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw\" (UID: \"e8480250-bbbd-44f3-a690-b5d2cb97efda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"
Oct 30 00:26:08.263423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.263385    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqfcj\" (UniqueName: \"kubernetes.io/projected/e8480250-bbbd-44f3-a690-b5d2cb97efda-kube-api-access-cqfcj\") pod \"payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw\" (UID: \"e8480250-bbbd-44f3-a690-b5d2cb97efda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"
Oct 30 00:26:08.352241 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.352157    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"
Oct 30 00:26:08.659086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.659055    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"]
Oct 30 00:26:08.660500 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:26:08.660469    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8480250_bbbd_44f3_a690_b5d2cb97efda.slice/crio-48b8500ee75318a8c82b407572f867a2f792f84746f963adc8902c692043a46c WatchSource:0}: Error finding container 48b8500ee75318a8c82b407572f867a2f792f84746f963adc8902c692043a46c: Status 404 returned error can't find the container with id 48b8500ee75318a8c82b407572f867a2f792f84746f963adc8902c692043a46c
Oct 30 00:26:08.894327 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.894277    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw" event={"ID":"e8480250-bbbd-44f3-a690-b5d2cb97efda","Type":"ContainerStarted","Data":"fcb0962e3c8daa737e8405b3cca6729f57b63aaede9280bcdaba3f3cf46aa5bb"}
Oct 30 00:26:08.894533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.894344    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw" event={"ID":"e8480250-bbbd-44f3-a690-b5d2cb97efda","Type":"ContainerStarted","Data":"48b8500ee75318a8c82b407572f867a2f792f84746f963adc8902c692043a46c"}
Oct 30 00:26:08.894914 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.894878    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"
Oct 30 00:26:08.935962 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:08.935904    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw" podStartSLOduration=0.93588435 podStartE2EDuration="935.88435ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:08.931249577 +0000 UTC m=+1278.456543453" watchObservedRunningTime="2026-10-30 00:26:08.93588435 +0000 UTC m=+1278.461178249"
Oct 30 00:26:09.233247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.233218    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:26:09.359857 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.359801    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-data\") pod \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") "
Oct 30 00:26:09.360021 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.359863    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-credential-socket\") pod \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") "
Oct 30 00:26:09.360021 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.359932    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-podinfo\") pod \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") "
Oct 30 00:26:09.360021 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.359991    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-token\" (UniqueName: \"kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-token\") pod \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") "
Oct 30 00:26:09.360246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.360062    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-socket\") pod \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") "
Oct 30 00:26:09.360246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.360109    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istiod-ca-cert\") pod \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") "
Oct 30 00:26:09.360351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.360206    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-envoy\") pod \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") "
Oct 30 00:26:09.360426 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.360402    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-certs\") pod \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") "
Oct 30 00:26:09.360554 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.360529    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-kube-api-access-f8gbt\" (UniqueName: \"kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-kube-api-access-f8gbt\") pod \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\" (UID: \"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5\") "
Oct 30 00:26:09.362298 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.362266    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-credential-socket" pod "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" (UID: "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:26:09.362409 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.362295    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istiod-ca-cert" pod "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" (UID: "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:26:09.362477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.362460    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-data" pod "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" (UID: "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:26:09.362609 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.362590    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-socket" pod "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" (UID: "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:26:09.363006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.362981    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-certs" pod "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" (UID: "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:26:09.364541 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.364515    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-kube-api-access-f8gbt" pod "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" (UID: "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5"). InnerVolumeSpecName "kube-api-access-f8gbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:26:09.365129 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.364997    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-podinfo" pod "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" (UID: "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:26:09.365364 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.365338    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-token" pod "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" (UID: "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:26:09.366510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.366484    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-envoy" pod "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" (UID: "f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:26:09.463637 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.463549    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:09.463637 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.463594    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:09.463637 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.463610    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:09.463637 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.463624    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:09.463637 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.463635    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:09.463931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.463648    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:09.463931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.463661    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:09.463931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.463673    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:09.463931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.463684    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8gbt\" (UniqueName: \"kubernetes.io/projected/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5-kube-api-access-f8gbt\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:09.948009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.947974    4441 generic.go:356] "Generic (PLEG): container finished" podID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerID="e742d6d453a430097636f549e1de20335941b668c586f7d071b1969e85981e89" exitCode=0
Oct 30 00:26:09.949439 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.949417    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"
Oct 30 00:26:09.952955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.952876    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" event={"ID":"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5","Type":"ContainerDied","Data":"e742d6d453a430097636f549e1de20335941b668c586f7d071b1969e85981e89"}
Oct 30 00:26:09.953082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.952966    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s" event={"ID":"f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5","Type":"ContainerDied","Data":"1ce6d40a2fc33f6fc2759444d89ee7ee34a9afab1214f5c8bc3b732b718ee149"}
Oct 30 00:26:09.953082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.953000    4441 scope.go:122] "RemoveContainer" containerID="e742d6d453a430097636f549e1de20335941b668c586f7d071b1969e85981e89"
Oct 30 00:26:09.999810 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:09.999700    4441 scope.go:122] "RemoveContainer" containerID="31080c30ea6d0f0fbca5e10172fc2bbdb3f4bddd766d5a78c86fcc66e5b188b9"
Oct 30 00:26:10.008966 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:10.008938    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"]
Oct 30 00:26:10.014986 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:10.014944    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0847e8c0-openshift-default-5d4b75b685-2s56s"]
Oct 30 00:26:10.020462 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:10.020443    4441 scope.go:122] "RemoveContainer" containerID="e742d6d453a430097636f549e1de20335941b668c586f7d071b1969e85981e89"
Oct 30 00:26:10.020770 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:26:10.020747    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e742d6d453a430097636f549e1de20335941b668c586f7d071b1969e85981e89\": container with ID starting with e742d6d453a430097636f549e1de20335941b668c586f7d071b1969e85981e89 not found: ID does not exist" containerID="e742d6d453a430097636f549e1de20335941b668c586f7d071b1969e85981e89"
Oct 30 00:26:10.020854 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:10.020777    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e742d6d453a430097636f549e1de20335941b668c586f7d071b1969e85981e89"} err="failed to get container status \"e742d6d453a430097636f549e1de20335941b668c586f7d071b1969e85981e89\": rpc error: code = NotFound desc = could not find container \"e742d6d453a430097636f549e1de20335941b668c586f7d071b1969e85981e89\": container with ID starting with e742d6d453a430097636f549e1de20335941b668c586f7d071b1969e85981e89 not found: ID does not exist"
Oct 30 00:26:10.020854 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:10.020797    4441 scope.go:122] "RemoveContainer" containerID="31080c30ea6d0f0fbca5e10172fc2bbdb3f4bddd766d5a78c86fcc66e5b188b9"
Oct 30 00:26:10.021103 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:26:10.021085    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31080c30ea6d0f0fbca5e10172fc2bbdb3f4bddd766d5a78c86fcc66e5b188b9\": container with ID starting with 31080c30ea6d0f0fbca5e10172fc2bbdb3f4bddd766d5a78c86fcc66e5b188b9 not found: ID does not exist" containerID="31080c30ea6d0f0fbca5e10172fc2bbdb3f4bddd766d5a78c86fcc66e5b188b9"
Oct 30 00:26:10.021174 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:10.021106    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31080c30ea6d0f0fbca5e10172fc2bbdb3f4bddd766d5a78c86fcc66e5b188b9"} err="failed to get container status \"31080c30ea6d0f0fbca5e10172fc2bbdb3f4bddd766d5a78c86fcc66e5b188b9\": rpc error: code = NotFound desc = could not find container \"31080c30ea6d0f0fbca5e10172fc2bbdb3f4bddd766d5a78c86fcc66e5b188b9\": container with ID starting with 31080c30ea6d0f0fbca5e10172fc2bbdb3f4bddd766d5a78c86fcc66e5b188b9 not found: ID does not exist"
Oct 30 00:26:10.612626 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:10.612597    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" path="/var/lib/kubelet/pods/f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5/volumes"
Oct 30 00:26:13.983047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:13.973785    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"]
Oct 30 00:26:13.983047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:13.974767    4441 cpu_manager.go:414] "containerMap: removing container" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy"
Oct 30 00:26:13.983047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:13.974789    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy"
Oct 30 00:26:13.983047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:13.975008    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy"
Oct 30 00:26:13.983047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:13.975050    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy"
Oct 30 00:26:14.122047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:14.114204    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"
Oct 30 00:26:14.130757 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:14.130732    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6cfcf7\""
Oct 30 00:26:14.134544 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:14.134514    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"]
Oct 30 00:26:14.140296 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:14.140250    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6cfcf7-dockercfg-gdskm\""
Oct 30 00:26:14.219901 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:14.219867    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppfkk\" (UniqueName: \"kubernetes.io/projected/4bb99ba7-cf69-456b-8384-57ef12a418be-kube-api-access-ppfkk\") pod \"payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp\" (UID: \"4bb99ba7-cf69-456b-8384-57ef12a418be\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"
Oct 30 00:26:14.220114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:14.219913    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb99ba7-cf69-456b-8384-57ef12a418be-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp\" (UID: \"4bb99ba7-cf69-456b-8384-57ef12a418be\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"
Oct 30 00:26:14.322362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:14.322274    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppfkk\" (UniqueName: \"kubernetes.io/projected/4bb99ba7-cf69-456b-8384-57ef12a418be-kube-api-access-ppfkk\") pod \"payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp\" (UID: \"4bb99ba7-cf69-456b-8384-57ef12a418be\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"
Oct 30 00:26:14.322362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:14.322319    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb99ba7-cf69-456b-8384-57ef12a418be-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp\" (UID: \"4bb99ba7-cf69-456b-8384-57ef12a418be\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"
Oct 30 00:26:14.323221 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:14.323189    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb99ba7-cf69-456b-8384-57ef12a418be-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp\" (UID: \"4bb99ba7-cf69-456b-8384-57ef12a418be\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"
Oct 30 00:26:14.343637 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:14.343599    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppfkk\" (UniqueName: \"kubernetes.io/projected/4bb99ba7-cf69-456b-8384-57ef12a418be-kube-api-access-ppfkk\") pod \"payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp\" (UID: \"4bb99ba7-cf69-456b-8384-57ef12a418be\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"
Oct 30 00:26:14.497691 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:14.497651    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"
Oct 30 00:26:14.808767 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:26:14.808718    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb99ba7_cf69_456b_8384_57ef12a418be.slice/crio-c6cf38b202b665ec49889cec6735a9e803bcaa234625c43ca5d59d076ba691c9 WatchSource:0}: Error finding container c6cf38b202b665ec49889cec6735a9e803bcaa234625c43ca5d59d076ba691c9: Status 404 returned error can't find the container with id c6cf38b202b665ec49889cec6735a9e803bcaa234625c43ca5d59d076ba691c9
Oct 30 00:26:14.809916 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:14.809537    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"]
Oct 30 00:26:15.200766 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:15.200686    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp" event={"ID":"4bb99ba7-cf69-456b-8384-57ef12a418be","Type":"ContainerStarted","Data":"524abb562cb4d2451040f24a2ef357dfe130925c4f54001a3338b92974de80ee"}
Oct 30 00:26:15.200766 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:15.200737    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp" event={"ID":"4bb99ba7-cf69-456b-8384-57ef12a418be","Type":"ContainerStarted","Data":"c6cf38b202b665ec49889cec6735a9e803bcaa234625c43ca5d59d076ba691c9"}
Oct 30 00:26:15.201696 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:15.201674    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"
Oct 30 00:26:15.238611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:15.238557    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp" podStartSLOduration=2.238542872 podStartE2EDuration="2.238542872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:15.235940272 +0000 UTC m=+1284.761234151" watchObservedRunningTime="2026-10-30 00:26:15.238542872 +0000 UTC m=+1284.763836792"
Oct 30 00:26:16.209263 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:16.209227    4441 generic.go:356] "Generic (PLEG): container finished" podID="c554ae97-8f0b-4bf9-bf77-6ac3bfb42575" containerID="e6f7b78f32afa089089f3b5d7f9ab99c9e81cf084bcf926d7df5881503abf659" exitCode=1
Oct 30 00:26:16.209712 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:16.209302    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2" event={"ID":"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575","Type":"ContainerDied","Data":"e6f7b78f32afa089089f3b5d7f9ab99c9e81cf084bcf926d7df5881503abf659"}
Oct 30 00:26:16.209764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:16.209732    4441 scope.go:122] "RemoveContainer" containerID="e6f7b78f32afa089089f3b5d7f9ab99c9e81cf084bcf926d7df5881503abf659"
Oct 30 00:26:16.801418 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:16.801380    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"]
Oct 30 00:26:16.802354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:16.802326    4441 cpu_manager.go:414] "containerMap: removing container" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy"
Oct 30 00:26:16.802354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:16.802347    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b0ff99-94b2-4f27-8ff8-156c5bfe92b5" containerName="istio-proxy"
Oct 30 00:26:16.806598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:16.806578    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"
Oct 30 00:26:16.822478 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:16.822445    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"]
Oct 30 00:26:16.827583 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:16.827562    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-bb16a8\""
Oct 30 00:26:16.827583 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:16.827574    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-bb16a8-dockercfg-6jdnq\""
Oct 30 00:26:17.038559 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.038513    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663f591e-ad15-44c2-9983-5464126875c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm\" (UID: \"663f591e-ad15-44c2-9983-5464126875c2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"
Oct 30 00:26:17.038731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.038692    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6v6p\" (UniqueName: \"kubernetes.io/projected/663f591e-ad15-44c2-9983-5464126875c2-kube-api-access-b6v6p\") pod \"payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm\" (UID: \"663f591e-ad15-44c2-9983-5464126875c2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"
Oct 30 00:26:17.058092 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.049890    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"]
Oct 30 00:26:17.060091 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.059219    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"]
Oct 30 00:26:17.060091 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.059335    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"
Oct 30 00:26:17.142959 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.141915    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663f591e-ad15-44c2-9983-5464126875c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm\" (UID: \"663f591e-ad15-44c2-9983-5464126875c2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"
Oct 30 00:26:17.142959 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.142140    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q6hb\" (UniqueName: \"kubernetes.io/projected/089e7957-fa4a-4bf4-b457-28af567838fd-kube-api-access-4q6hb\") pod \"payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc\" (UID: \"089e7957-fa4a-4bf4-b457-28af567838fd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"
Oct 30 00:26:17.142959 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.142191    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6v6p\" (UniqueName: \"kubernetes.io/projected/663f591e-ad15-44c2-9983-5464126875c2-kube-api-access-b6v6p\") pod \"payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm\" (UID: \"663f591e-ad15-44c2-9983-5464126875c2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"
Oct 30 00:26:17.142959 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.142370    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089e7957-fa4a-4bf4-b457-28af567838fd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc\" (UID: \"089e7957-fa4a-4bf4-b457-28af567838fd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"
Oct 30 00:26:17.143318 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.143278    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663f591e-ad15-44c2-9983-5464126875c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm\" (UID: \"663f591e-ad15-44c2-9983-5464126875c2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"
Oct 30 00:26:17.164974 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.164942    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6v6p\" (UniqueName: \"kubernetes.io/projected/663f591e-ad15-44c2-9983-5464126875c2-kube-api-access-b6v6p\") pod \"payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm\" (UID: \"663f591e-ad15-44c2-9983-5464126875c2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"
Oct 30 00:26:17.221282 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.221240    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2" event={"ID":"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575","Type":"ContainerStarted","Data":"dbbf68baf0a49135a7b67b62cfcdfceeba4f33a8d0ed63631a36a3b0a8511387"}
Oct 30 00:26:17.243897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.243863    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q6hb\" (UniqueName: \"kubernetes.io/projected/089e7957-fa4a-4bf4-b457-28af567838fd-kube-api-access-4q6hb\") pod \"payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc\" (UID: \"089e7957-fa4a-4bf4-b457-28af567838fd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"
Oct 30 00:26:17.244159 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.244058    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089e7957-fa4a-4bf4-b457-28af567838fd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc\" (UID: \"089e7957-fa4a-4bf4-b457-28af567838fd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"
Oct 30 00:26:17.245088 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.245058    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089e7957-fa4a-4bf4-b457-28af567838fd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc\" (UID: \"089e7957-fa4a-4bf4-b457-28af567838fd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"
Oct 30 00:26:17.269710 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.269680    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q6hb\" (UniqueName: \"kubernetes.io/projected/089e7957-fa4a-4bf4-b457-28af567838fd-kube-api-access-4q6hb\") pod \"payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc\" (UID: \"089e7957-fa4a-4bf4-b457-28af567838fd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"
Oct 30 00:26:17.383680 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.383592    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"
Oct 30 00:26:17.424044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.422988    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"
Oct 30 00:26:17.719322 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.719253    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"]
Oct 30 00:26:17.883445 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:17.883350    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"]
Oct 30 00:26:17.915056 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:26:17.902225    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663f591e_ad15_44c2_9983_5464126875c2.slice/crio-7b809d52c6b8b2bb331dbca1c3cde680f0406244c584e3abea10bbeab63c22d4 WatchSource:0}: Error finding container 7b809d52c6b8b2bb331dbca1c3cde680f0406244c584e3abea10bbeab63c22d4: Status 404 returned error can't find the container with id 7b809d52c6b8b2bb331dbca1c3cde680f0406244c584e3abea10bbeab63c22d4
Oct 30 00:26:18.237137 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:18.234351    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm" event={"ID":"663f591e-ad15-44c2-9983-5464126875c2","Type":"ContainerStarted","Data":"ae23571b87b0bc2fdc6173febbb926b852e4a1d2a05da9f85d913e5538ac1a37"}
Oct 30 00:26:18.237137 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:18.234429    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm" event={"ID":"663f591e-ad15-44c2-9983-5464126875c2","Type":"ContainerStarted","Data":"7b809d52c6b8b2bb331dbca1c3cde680f0406244c584e3abea10bbeab63c22d4"}
Oct 30 00:26:18.237137 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:18.234568    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"
Oct 30 00:26:18.237137 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:18.236550    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc" event={"ID":"089e7957-fa4a-4bf4-b457-28af567838fd","Type":"ContainerStarted","Data":"8069a8669c576dc257314da70fada451665f14046e8f916625ce4f1eeb15296e"}
Oct 30 00:26:18.237137 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:18.236587    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc" event={"ID":"089e7957-fa4a-4bf4-b457-28af567838fd","Type":"ContainerStarted","Data":"c6ea9a9c37bf4bd4dadc4fc6b81b9c2a381d9f4bb3cb7fcc9e2e2a8f93f22fce"}
Oct 30 00:26:18.237794 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:18.237143    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"
Oct 30 00:26:18.308842 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:18.308784    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm" podStartSLOduration=2.30875612 podStartE2EDuration="2.30875612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:18.272177007 +0000 UTC m=+1287.797470885" watchObservedRunningTime="2026-10-30 00:26:18.30875612 +0000 UTC m=+1287.834050001"
Oct 30 00:26:18.310743 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:18.310701    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc" podStartSLOduration=2.310692593 podStartE2EDuration="2.310692593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:18.306088279 +0000 UTC m=+1287.831382159" watchObservedRunningTime="2026-10-30 00:26:18.310692593 +0000 UTC m=+1287.835986467"
Oct 30 00:26:19.954697 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:19.954667    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"
Oct 30 00:26:25.017535 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.017495    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"]
Oct 30 00:26:25.022860 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.022834    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"
Oct 30 00:26:25.045468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.045441    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-93016b\""
Oct 30 00:26:25.045468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.045442    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-93016b-dockercfg-5mgqr\""
Oct 30 00:26:25.050689 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.049931    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"]
Oct 30 00:26:25.171104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.169038    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e91820-48d8-494c-a1b4-d18db107bfe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj\" (UID: \"a9e91820-48d8-494c-a1b4-d18db107bfe8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"
Oct 30 00:26:25.171104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.169188    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvkb4\" (UniqueName: \"kubernetes.io/projected/a9e91820-48d8-494c-a1b4-d18db107bfe8-kube-api-access-bvkb4\") pod \"payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj\" (UID: \"a9e91820-48d8-494c-a1b4-d18db107bfe8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"
Oct 30 00:26:25.284728 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.284628    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkb4\" (UniqueName: \"kubernetes.io/projected/a9e91820-48d8-494c-a1b4-d18db107bfe8-kube-api-access-bvkb4\") pod \"payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj\" (UID: \"a9e91820-48d8-494c-a1b4-d18db107bfe8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"
Oct 30 00:26:25.284934 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.284910    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e91820-48d8-494c-a1b4-d18db107bfe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj\" (UID: \"a9e91820-48d8-494c-a1b4-d18db107bfe8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"
Oct 30 00:26:25.285976 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.285946    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e91820-48d8-494c-a1b4-d18db107bfe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj\" (UID: \"a9e91820-48d8-494c-a1b4-d18db107bfe8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"
Oct 30 00:26:25.305650 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.305498    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"]
Oct 30 00:26:25.344045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.340955    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkb4\" (UniqueName: \"kubernetes.io/projected/a9e91820-48d8-494c-a1b4-d18db107bfe8-kube-api-access-bvkb4\") pod \"payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj\" (UID: \"a9e91820-48d8-494c-a1b4-d18db107bfe8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"
Oct 30 00:26:25.344045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.341310    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"
Oct 30 00:26:25.363247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.363219    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"]
Oct 30 00:26:25.363407 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.363335    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"
Oct 30 00:26:25.488784 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.488719    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4c26\" (UniqueName: \"kubernetes.io/projected/55345590-08a0-4f55-b6f5-4768c149ad8b-kube-api-access-v4c26\") pod \"payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8\" (UID: \"55345590-08a0-4f55-b6f5-4768c149ad8b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"
Oct 30 00:26:25.489141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.489016    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55345590-08a0-4f55-b6f5-4768c149ad8b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8\" (UID: \"55345590-08a0-4f55-b6f5-4768c149ad8b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"
Oct 30 00:26:25.591257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.591143    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55345590-08a0-4f55-b6f5-4768c149ad8b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8\" (UID: \"55345590-08a0-4f55-b6f5-4768c149ad8b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"
Oct 30 00:26:25.591423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.591365    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4c26\" (UniqueName: \"kubernetes.io/projected/55345590-08a0-4f55-b6f5-4768c149ad8b-kube-api-access-v4c26\") pod \"payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8\" (UID: \"55345590-08a0-4f55-b6f5-4768c149ad8b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"
Oct 30 00:26:25.591947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.591920    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55345590-08a0-4f55-b6f5-4768c149ad8b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8\" (UID: \"55345590-08a0-4f55-b6f5-4768c149ad8b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"
Oct 30 00:26:25.612320 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.612295    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4c26\" (UniqueName: \"kubernetes.io/projected/55345590-08a0-4f55-b6f5-4768c149ad8b-kube-api-access-v4c26\") pod \"payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8\" (UID: \"55345590-08a0-4f55-b6f5-4768c149ad8b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"
Oct 30 00:26:25.671304 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.671208    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"]
Oct 30 00:26:25.674771 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:26:25.674745    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e91820_48d8_494c_a1b4_d18db107bfe8.slice/crio-97c381dd8ba7d72f3cd86b565d0de673de65c74bb37adcdfb5625db1043f018f WatchSource:0}: Error finding container 97c381dd8ba7d72f3cd86b565d0de673de65c74bb37adcdfb5625db1043f018f: Status 404 returned error can't find the container with id 97c381dd8ba7d72f3cd86b565d0de673de65c74bb37adcdfb5625db1043f018f
Oct 30 00:26:25.714225 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:25.714204    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"
Oct 30 00:26:26.027885 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:26.027853    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"]
Oct 30 00:26:26.028725 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:26:26.028690    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55345590_08a0_4f55_b6f5_4768c149ad8b.slice/crio-361cbbac0b7f11560134ecb3c0d8a1ec5aff04e485bdcc34cfaaafbc2717f11f WatchSource:0}: Error finding container 361cbbac0b7f11560134ecb3c0d8a1ec5aff04e485bdcc34cfaaafbc2717f11f: Status 404 returned error can't find the container with id 361cbbac0b7f11560134ecb3c0d8a1ec5aff04e485bdcc34cfaaafbc2717f11f
Oct 30 00:26:26.346947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:26.346847    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj" event={"ID":"a9e91820-48d8-494c-a1b4-d18db107bfe8","Type":"ContainerStarted","Data":"274f0217cb55a4c4faf9e027489ee4e5a32ffcea0c447dea6eff4cc6e1bd3c38"}
Oct 30 00:26:26.346947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:26.346920    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj" event={"ID":"a9e91820-48d8-494c-a1b4-d18db107bfe8","Type":"ContainerStarted","Data":"97c381dd8ba7d72f3cd86b565d0de673de65c74bb37adcdfb5625db1043f018f"}
Oct 30 00:26:26.356446 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:26.356415    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"
Oct 30 00:26:26.361638 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:26.361605    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8" event={"ID":"55345590-08a0-4f55-b6f5-4768c149ad8b","Type":"ContainerStarted","Data":"2a103d7f137d0c469e0fe658b647888d27f0514e0dac62ac39767bb349fe67eb"}
Oct 30 00:26:26.361787 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:26.361659    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8" event={"ID":"55345590-08a0-4f55-b6f5-4768c149ad8b","Type":"ContainerStarted","Data":"361cbbac0b7f11560134ecb3c0d8a1ec5aff04e485bdcc34cfaaafbc2717f11f"}
Oct 30 00:26:26.361787 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:26.361761    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"
Oct 30 00:26:26.394608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:26.394545    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj" podStartSLOduration=2.394525248 podStartE2EDuration="2.394525248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:26.391991079 +0000 UTC m=+1295.917284969" watchObservedRunningTime="2026-10-30 00:26:26.394525248 +0000 UTC m=+1295.919819128"
Oct 30 00:26:26.434276 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:26.434214    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8" podStartSLOduration=1.43419618 podStartE2EDuration="1.43419618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:26.432777117 +0000 UTC m=+1295.958070995" watchObservedRunningTime="2026-10-30 00:26:26.43419618 +0000 UTC m=+1295.959490058"
Oct 30 00:26:27.222129 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:27.222094    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"
Oct 30 00:26:27.223902 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:27.223877    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"
Oct 30 00:26:27.225087 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:27.225064    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"
Oct 30 00:26:29.250492 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.250459    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"]
Oct 30 00:26:29.255847 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.255819    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"
Oct 30 00:26:29.256014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.255922    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"
Oct 30 00:26:29.276810 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.276778    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"]
Oct 30 00:26:29.278695 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.278671    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f649b0\""
Oct 30 00:26:29.340340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.340305    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"]
Oct 30 00:26:29.353047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.350795    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f649b0-dockercfg-gn6h8\""
Oct 30 00:26:29.390075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.382624    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4906b857-b547-40b4-bf72-8829b532e404-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft\" (UID: \"4906b857-b547-40b4-bf72-8829b532e404\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"
Oct 30 00:26:29.390075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.382792    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msc2x\" (UniqueName: \"kubernetes.io/projected/4906b857-b547-40b4-bf72-8829b532e404-kube-api-access-msc2x\") pod \"payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft\" (UID: \"4906b857-b547-40b4-bf72-8829b532e404\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"
Oct 30 00:26:29.432506 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.423802    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"]
Oct 30 00:26:29.432506 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.423913    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"
Oct 30 00:26:29.521570 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.516002    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4906b857-b547-40b4-bf72-8829b532e404-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft\" (UID: \"4906b857-b547-40b4-bf72-8829b532e404\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"
Oct 30 00:26:29.521570 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.516217    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msc2x\" (UniqueName: \"kubernetes.io/projected/4906b857-b547-40b4-bf72-8829b532e404-kube-api-access-msc2x\") pod \"payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft\" (UID: \"4906b857-b547-40b4-bf72-8829b532e404\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"
Oct 30 00:26:29.564098 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.562842    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4906b857-b547-40b4-bf72-8829b532e404-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft\" (UID: \"4906b857-b547-40b4-bf72-8829b532e404\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"
Oct 30 00:26:29.566810 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.566744    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msc2x\" (UniqueName: \"kubernetes.io/projected/4906b857-b547-40b4-bf72-8829b532e404-kube-api-access-msc2x\") pod \"payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft\" (UID: \"4906b857-b547-40b4-bf72-8829b532e404\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"
Oct 30 00:26:29.581039 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.580923    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"
Oct 30 00:26:29.619046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.618217    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p95qt\" (UniqueName: \"kubernetes.io/projected/af5d056f-dec9-4092-999d-60b735e96e06-kube-api-access-p95qt\") pod \"payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f\" (UID: \"af5d056f-dec9-4092-999d-60b735e96e06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"
Oct 30 00:26:29.619046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.618323    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5d056f-dec9-4092-999d-60b735e96e06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f\" (UID: \"af5d056f-dec9-4092-999d-60b735e96e06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"
Oct 30 00:26:29.735923 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.735831    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p95qt\" (UniqueName: \"kubernetes.io/projected/af5d056f-dec9-4092-999d-60b735e96e06-kube-api-access-p95qt\") pod \"payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f\" (UID: \"af5d056f-dec9-4092-999d-60b735e96e06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"
Oct 30 00:26:29.738120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.736231    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5d056f-dec9-4092-999d-60b735e96e06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f\" (UID: \"af5d056f-dec9-4092-999d-60b735e96e06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"
Oct 30 00:26:29.738120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.737320    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5d056f-dec9-4092-999d-60b735e96e06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f\" (UID: \"af5d056f-dec9-4092-999d-60b735e96e06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"
Oct 30 00:26:29.765155 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.765078    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p95qt\" (UniqueName: \"kubernetes.io/projected/af5d056f-dec9-4092-999d-60b735e96e06-kube-api-access-p95qt\") pod \"payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f\" (UID: \"af5d056f-dec9-4092-999d-60b735e96e06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"
Oct 30 00:26:29.908670 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.908350    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"
Oct 30 00:26:29.965621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:29.965590    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"]
Oct 30 00:26:30.304118 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:30.296461    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"
Oct 30 00:26:30.540107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:30.524142    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft" event={"ID":"4906b857-b547-40b4-bf72-8829b532e404","Type":"ContainerStarted","Data":"86bea419a5b90ddd2483562dc37d3d4d0c991829a1b4f025ff6c2852eb24bcdf"}
Oct 30 00:26:30.886081 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:30.885978    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"]
Oct 30 00:26:31.538089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:31.538011    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f" event={"ID":"af5d056f-dec9-4092-999d-60b735e96e06","Type":"ContainerStarted","Data":"a4066e887ee99a3e82988b505af53db4b0797d2632301bc2b61603c30a991f5d"}
Oct 30 00:26:31.538783 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:31.538756    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f" event={"ID":"af5d056f-dec9-4092-999d-60b735e96e06","Type":"ContainerStarted","Data":"1f3144b419f70aca72cb266de4fff73e909533b31c609c29d99670db851402eb"}
Oct 30 00:26:31.539240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:31.538917    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"
Oct 30 00:26:31.541188 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:31.541160    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft" event={"ID":"4906b857-b547-40b4-bf72-8829b532e404","Type":"ContainerStarted","Data":"325e06ea497a7b1dfaa07d7d1a88d686f13644321ae0b01ecec8d796b1a3c029"}
Oct 30 00:26:31.541305 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:31.541238    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"
Oct 30 00:26:31.578511 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:31.578447    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f" podStartSLOduration=2.578428688 podStartE2EDuration="2.578428688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:31.573567928 +0000 UTC m=+1301.098861805" watchObservedRunningTime="2026-10-30 00:26:31.578428688 +0000 UTC m=+1301.103722571"
Oct 30 00:26:31.612073 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:31.612005    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft" podStartSLOduration=2.611989915 podStartE2EDuration="2.611989915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:31.608064261 +0000 UTC m=+1301.133358135" watchObservedRunningTime="2026-10-30 00:26:31.611989915 +0000 UTC m=+1301.137283791"
Oct 30 00:26:33.900998 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:26:33.900958    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:26:36.276610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.276570    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"]
Oct 30 00:26:36.277275 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.276846    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z" podUID="30206a75-65df-4bc5-893f-5f4dc29a650c" containerName="istio-proxy" containerID="cri-o://7f1ef0950c85c2e928dffdea2aaa550deeb13283c7411c2c63e2371f421b21f1" gracePeriod=30
Oct 30 00:26:36.625755 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.625673    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"]
Oct 30 00:26:36.630847 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.630824    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"
Oct 30 00:26:36.643907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.643881    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a15230\""
Oct 30 00:26:36.644074 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.644009    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a15230-dockercfg-225nr\""
Oct 30 00:26:36.655161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.655138    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"]
Oct 30 00:26:36.778122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.778086    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"]
Oct 30 00:26:36.810658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.810617    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn4fb\" (UniqueName: \"kubernetes.io/projected/a95c0bd0-aeea-40db-923b-acddedcac7c6-kube-api-access-xn4fb\") pod \"payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7\" (UID: \"a95c0bd0-aeea-40db-923b-acddedcac7c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"
Oct 30 00:26:36.825067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.820812    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95c0bd0-aeea-40db-923b-acddedcac7c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7\" (UID: \"a95c0bd0-aeea-40db-923b-acddedcac7c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"
Oct 30 00:26:36.859110 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.839594    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"]
Oct 30 00:26:36.859110 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.839717    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"
Oct 30 00:26:36.934960 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.934865    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95c0bd0-aeea-40db-923b-acddedcac7c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7\" (UID: \"a95c0bd0-aeea-40db-923b-acddedcac7c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"
Oct 30 00:26:36.934960 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.934944    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv\" (UID: \"cbd1c416-58d1-4259-aa8c-bbf63b887e3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"
Oct 30 00:26:36.935232 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.935010    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn4fb\" (UniqueName: \"kubernetes.io/projected/a95c0bd0-aeea-40db-923b-acddedcac7c6-kube-api-access-xn4fb\") pod \"payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7\" (UID: \"a95c0bd0-aeea-40db-923b-acddedcac7c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"
Oct 30 00:26:36.935232 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.935045    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfz68\" (UniqueName: \"kubernetes.io/projected/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-kube-api-access-jfz68\") pod \"payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv\" (UID: \"cbd1c416-58d1-4259-aa8c-bbf63b887e3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"
Oct 30 00:26:36.936208 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.936169    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95c0bd0-aeea-40db-923b-acddedcac7c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7\" (UID: \"a95c0bd0-aeea-40db-923b-acddedcac7c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"
Oct 30 00:26:36.956620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:36.956585    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn4fb\" (UniqueName: \"kubernetes.io/projected/a95c0bd0-aeea-40db-923b-acddedcac7c6-kube-api-access-xn4fb\") pod \"payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7\" (UID: \"a95c0bd0-aeea-40db-923b-acddedcac7c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"
Oct 30 00:26:37.036336 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:37.036300    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfz68\" (UniqueName: \"kubernetes.io/projected/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-kube-api-access-jfz68\") pod \"payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv\" (UID: \"cbd1c416-58d1-4259-aa8c-bbf63b887e3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"
Oct 30 00:26:37.036921 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:37.036886    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv\" (UID: \"cbd1c416-58d1-4259-aa8c-bbf63b887e3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"
Oct 30 00:26:37.038131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:37.038109    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv\" (UID: \"cbd1c416-58d1-4259-aa8c-bbf63b887e3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"
Oct 30 00:26:37.058774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:37.058742    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfz68\" (UniqueName: \"kubernetes.io/projected/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-kube-api-access-jfz68\") pod \"payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv\" (UID: \"cbd1c416-58d1-4259-aa8c-bbf63b887e3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"
Oct 30 00:26:37.164422 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:37.164374    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"
Oct 30 00:26:37.250055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:37.249459    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"
Oct 30 00:26:37.405538 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:37.392717    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"
Oct 30 00:26:37.477962 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:37.477925    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"]
Oct 30 00:26:37.479396 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:26:37.479358    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd1c416_58d1_4259_aa8c_bbf63b887e3f.slice/crio-08867c4657969ffa1718aea26cd081d9a17d0b7142b4ff9a12ff45a3dd905552 WatchSource:0}: Error finding container 08867c4657969ffa1718aea26cd081d9a17d0b7142b4ff9a12ff45a3dd905552: Status 404 returned error can't find the container with id 08867c4657969ffa1718aea26cd081d9a17d0b7142b4ff9a12ff45a3dd905552
Oct 30 00:26:37.614471 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:37.614443    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"]
Oct 30 00:26:37.641754 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:37.641718    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv" event={"ID":"cbd1c416-58d1-4259-aa8c-bbf63b887e3f","Type":"ContainerStarted","Data":"3933e823187a61215d8e85ea5cbd89c045fe7d6f2e4a85dc480efd4ebbbf5ce4"}
Oct 30 00:26:37.641993 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:37.641967    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv" event={"ID":"cbd1c416-58d1-4259-aa8c-bbf63b887e3f","Type":"ContainerStarted","Data":"08867c4657969ffa1718aea26cd081d9a17d0b7142b4ff9a12ff45a3dd905552"}
Oct 30 00:26:37.643447 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:37.643426    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"
Oct 30 00:26:37.709836 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:37.709335    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv" podStartSLOduration=1.709314193 podStartE2EDuration="1.709314193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:37.708239339 +0000 UTC m=+1307.233533217" watchObservedRunningTime="2026-10-30 00:26:37.709314193 +0000 UTC m=+1307.234608071"
Oct 30 00:26:38.401603 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:38.401575    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"
Oct 30 00:26:38.655561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:38.651924    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7" event={"ID":"a95c0bd0-aeea-40db-923b-acddedcac7c6","Type":"ContainerStarted","Data":"eeba9c7b5ea54a648c3634cde22fa508fedf7232abb1732c4122946f6aeeb5c8"}
Oct 30 00:26:38.655561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:38.651985    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7" event={"ID":"a95c0bd0-aeea-40db-923b-acddedcac7c6","Type":"ContainerStarted","Data":"ea034cdaec13038b64011d7525ddd44de5b19297548d1f6dfc8d1feb8fe2f55e"}
Oct 30 00:26:38.655561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:38.652052    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"
Oct 30 00:26:38.690297 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:38.690233    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7" podStartSLOduration=2.690214817 podStartE2EDuration="2.690214817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:38.687139909 +0000 UTC m=+1308.212433787" watchObservedRunningTime="2026-10-30 00:26:38.690214817 +0000 UTC m=+1308.215508695"
Oct 30 00:26:41.843784 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:41.843472    4441 generic.go:356] "Generic (PLEG): container finished" podID="30206a75-65df-4bc5-893f-5f4dc29a650c" containerID="7f1ef0950c85c2e928dffdea2aaa550deeb13283c7411c2c63e2371f421b21f1" exitCode=0
Oct 30 00:26:41.843784 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:41.843714    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z" event={"ID":"30206a75-65df-4bc5-893f-5f4dc29a650c","Type":"ContainerDied","Data":"7f1ef0950c85c2e928dffdea2aaa550deeb13283c7411c2c63e2371f421b21f1"}
Oct 30 00:26:41.938871 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:41.938812    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:26:42.004354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.004095    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-socket\") pod \"30206a75-65df-4bc5-893f-5f4dc29a650c\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") "
Oct 30 00:26:42.004354 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.004342    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-credential-socket\") pod \"30206a75-65df-4bc5-893f-5f4dc29a650c\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") "
Oct 30 00:26:42.004668 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.004629    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-credential-socket" pod "30206a75-65df-4bc5-893f-5f4dc29a650c" (UID: "30206a75-65df-4bc5-893f-5f4dc29a650c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:26:42.004812 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.004658    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-socket" pod "30206a75-65df-4bc5-893f-5f4dc29a650c" (UID: "30206a75-65df-4bc5-893f-5f4dc29a650c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:26:42.004953 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.004927    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-podinfo\") pod \"30206a75-65df-4bc5-893f-5f4dc29a650c\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") "
Oct 30 00:26:42.005093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.005035    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-data\") pod \"30206a75-65df-4bc5-893f-5f4dc29a650c\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") "
Oct 30 00:26:42.005617 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.005152    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-kube-api-access-qgjtm\" (UniqueName: \"kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-kube-api-access-qgjtm\") pod \"30206a75-65df-4bc5-893f-5f4dc29a650c\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") "
Oct 30 00:26:42.005617 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.005250    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30206a75-65df-4bc5-893f-5f4dc29a650c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30206a75-65df-4bc5-893f-5f4dc29a650c-istiod-ca-cert\") pod \"30206a75-65df-4bc5-893f-5f4dc29a650c\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") "
Oct 30 00:26:42.005617 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.005287    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-token\" (UniqueName: \"kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-token\") pod \"30206a75-65df-4bc5-893f-5f4dc29a650c\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") "
Oct 30 00:26:42.005617 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.005352    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-certs\") pod \"30206a75-65df-4bc5-893f-5f4dc29a650c\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") "
Oct 30 00:26:42.006291 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.006110    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:42.006291 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.006136    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:42.006291 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.006259    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-certs" pod "30206a75-65df-4bc5-893f-5f4dc29a650c" (UID: "30206a75-65df-4bc5-893f-5f4dc29a650c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:26:42.006492 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.006348    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30206a75-65df-4bc5-893f-5f4dc29a650c-istiod-ca-cert" pod "30206a75-65df-4bc5-893f-5f4dc29a650c" (UID: "30206a75-65df-4bc5-893f-5f4dc29a650c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:26:42.006492 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.006439    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-data" pod "30206a75-65df-4bc5-893f-5f4dc29a650c" (UID: "30206a75-65df-4bc5-893f-5f4dc29a650c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:26:42.021248 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.021217    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-podinfo" pod "30206a75-65df-4bc5-893f-5f4dc29a650c" (UID: "30206a75-65df-4bc5-893f-5f4dc29a650c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:26:42.021374 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.021352    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-kube-api-access-qgjtm" pod "30206a75-65df-4bc5-893f-5f4dc29a650c" (UID: "30206a75-65df-4bc5-893f-5f4dc29a650c"). InnerVolumeSpecName "kube-api-access-qgjtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:26:42.021468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.021231    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-token" pod "30206a75-65df-4bc5-893f-5f4dc29a650c" (UID: "30206a75-65df-4bc5-893f-5f4dc29a650c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:26:42.108955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.106991    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-envoy\") pod \"30206a75-65df-4bc5-893f-5f4dc29a650c\" (UID: \"30206a75-65df-4bc5-893f-5f4dc29a650c\") "
Oct 30 00:26:42.108955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.108147    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:42.108955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.108175    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:42.108955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.108184    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgjtm\" (UniqueName: \"kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-kube-api-access-qgjtm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:42.108955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.108193    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30206a75-65df-4bc5-893f-5f4dc29a650c-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:42.108955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.108202    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:42.108955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.108209    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:42.112780 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.112749    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-envoy" pod "30206a75-65df-4bc5-893f-5f4dc29a650c" (UID: "30206a75-65df-4bc5-893f-5f4dc29a650c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:26:42.210058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.209972    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30206a75-65df-4bc5-893f-5f4dc29a650c-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:26:42.563147 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.563115    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"
Oct 30 00:26:42.852322 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.852250    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"
Oct 30 00:26:42.852322 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.852289    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z" event={"ID":"30206a75-65df-4bc5-893f-5f4dc29a650c","Type":"ContainerDied","Data":"62a83d46a4c3481de1e09eb9435a9f41574be998c3850b422aa7190866d6b3a8"}
Oct 30 00:26:42.852908 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.852375    4441 scope.go:122] "RemoveContainer" containerID="7f1ef0950c85c2e928dffdea2aaa550deeb13283c7411c2c63e2371f421b21f1"
Oct 30 00:26:42.854469 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.854443    4441 generic.go:356] "Generic (PLEG): container finished" podID="185b11f8-5a8a-4282-b9ae-53c9182d0668" containerID="14d881485fc61f9853d49e2b3997828bb05ba133934d3ee95ce1b6f4affe5f1c" exitCode=0
Oct 30 00:26:42.854562 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.854518    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd" event={"ID":"185b11f8-5a8a-4282-b9ae-53c9182d0668","Type":"ContainerDied","Data":"14d881485fc61f9853d49e2b3997828bb05ba133934d3ee95ce1b6f4affe5f1c"}
Oct 30 00:26:42.855051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.855010    4441 scope.go:122] "RemoveContainer" containerID="14d881485fc61f9853d49e2b3997828bb05ba133934d3ee95ce1b6f4affe5f1c"
Oct 30 00:26:42.856633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.856609    4441 generic.go:356] "Generic (PLEG): container finished" podID="88fc9fb9-507f-455a-93b1-6d158cc5896f" containerID="4f164873739d35d722624fae418237cd7367bcb2a002c2b37e828c17857d2e6f" exitCode=1
Oct 30 00:26:42.856758 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.856661    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-877hw" event={"ID":"88fc9fb9-507f-455a-93b1-6d158cc5896f","Type":"ContainerDied","Data":"4f164873739d35d722624fae418237cd7367bcb2a002c2b37e828c17857d2e6f"}
Oct 30 00:26:42.857070 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.857050    4441 scope.go:122] "RemoveContainer" containerID="4f164873739d35d722624fae418237cd7367bcb2a002c2b37e828c17857d2e6f"
Oct 30 00:26:42.962900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.962873    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-877hw"
Oct 30 00:26:42.980886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.980856    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"]
Oct 30 00:26:42.986783 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:42.986757    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-lsg6z"]
Oct 30 00:26:43.572102 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:43.572072    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"
Oct 30 00:26:43.868680 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:43.868590    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-877hw" event={"ID":"88fc9fb9-507f-455a-93b1-6d158cc5896f","Type":"ContainerStarted","Data":"207c3c796b8e3f4509e52917baa067b2082f743191d90f9e344b80d7311bff80"}
Oct 30 00:26:43.869161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:43.868869    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-877hw"
Oct 30 00:26:43.872389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:43.872352    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd" event={"ID":"185b11f8-5a8a-4282-b9ae-53c9182d0668","Type":"ContainerStarted","Data":"9875f4040bca38f3dc6b77f0f2474da776f1b1da620052e7205d97133c916c4f"}
Oct 30 00:26:43.872558 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:43.872541    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"
Oct 30 00:26:44.620167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:44.620130    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30206a75-65df-4bc5-893f-5f4dc29a650c" path="/var/lib/kubelet/pods/30206a75-65df-4bc5-893f-5f4dc29a650c/volumes"
Oct 30 00:26:45.979399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:45.979357    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"]
Oct 30 00:26:45.980391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:45.980355    4441 cpu_manager.go:414] "containerMap: removing container" podUID="30206a75-65df-4bc5-893f-5f4dc29a650c" containerName="istio-proxy"
Oct 30 00:26:45.980391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:45.980385    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30206a75-65df-4bc5-893f-5f4dc29a650c" containerName="istio-proxy"
Oct 30 00:26:45.980598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:45.980545    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="30206a75-65df-4bc5-893f-5f4dc29a650c" containerName="istio-proxy"
Oct 30 00:26:45.984563 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:45.984544    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.010453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.010428    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-93016b-serving-cert\""
Oct 30 00:26:46.011340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.011309    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"]
Oct 30 00:26:46.081473 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.081430    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.081473 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.081478    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-metrics-certs\") pod \"maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.081727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.081684    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.081785 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.081745    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvf6z\" (UniqueName: \"kubernetes.io/projected/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-kube-api-access-rvf6z\") pod \"maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.192233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.192196    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.192233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.192241    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-metrics-certs\") pod \"maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.194016 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.192379    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.194016 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.192490    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvf6z\" (UniqueName: \"kubernetes.io/projected/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-kube-api-access-rvf6z\") pod \"maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.194016 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.193359    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.195792 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.195765    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.196981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.196877    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-metrics-certs\") pod \"maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.217335 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.217188    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvf6z\" (UniqueName: \"kubernetes.io/projected/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-kube-api-access-rvf6z\") pod \"maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.307912 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.307801    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:46.658498 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:26:46.658463    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb1eed7_aaad_47b4_9bf7_005a5403f0af.slice/crio-01b3540961741a908fe17a43365b84f90bcc7ad3d5374ce1f87bb77f7f050e7e WatchSource:0}: Error finding container 01b3540961741a908fe17a43365b84f90bcc7ad3d5374ce1f87bb77f7f050e7e: Status 404 returned error can't find the container with id 01b3540961741a908fe17a43365b84f90bcc7ad3d5374ce1f87bb77f7f050e7e
Oct 30 00:26:46.658606 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.658502    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"]
Oct 30 00:26:46.902739 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:46.902701    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m" event={"ID":"bfb1eed7-aaad-47b4-9bf7-005a5403f0af","Type":"ContainerStarted","Data":"01b3540961741a908fe17a43365b84f90bcc7ad3d5374ce1f87bb77f7f050e7e"}
Oct 30 00:26:47.912753 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:47.912718    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m" event={"ID":"bfb1eed7-aaad-47b4-9bf7-005a5403f0af","Type":"ContainerStarted","Data":"46cfc9cb1947101b34936358cc57f92015e6560d69246247dd9f32de9fa14e3d"}
Oct 30 00:26:47.913160 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:47.912781    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:47.951872 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:47.951815    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m" podStartSLOduration=2.526989045 podStartE2EDuration="2.951801519s" totalImagesPullingTime="424.812474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:47.949582294 +0000 UTC m=+1317.474876171" watchObservedRunningTime="2026-10-30 00:26:47.951801519 +0000 UTC m=+1317.477095395"
Oct 30 00:26:49.661272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:49.661244    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"
Oct 30 00:26:49.661641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:49.661307    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"
Oct 30 00:26:53.926686 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:53.926658    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:26:54.884751 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:26:54.884718    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tmkmd"
Oct 30 00:27:00.242124 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.242089    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"]
Oct 30 00:27:00.248129 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.248108    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.263323 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.263290    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a15230-serving-cert\""
Oct 30 00:27:00.299936 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.299901    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"]
Oct 30 00:27:00.349319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.349213    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe107d75-925a-4bf8-99db-c497e80dd4b1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.349319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.349288    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.349572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.349360    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjwtz\" (UniqueName: \"kubernetes.io/projected/fe107d75-925a-4bf8-99db-c497e80dd4b1-kube-api-access-jjwtz\") pod \"maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.349572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.349434    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-metrics-certs\") pod \"maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.451195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.451163    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe107d75-925a-4bf8-99db-c497e80dd4b1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.451404 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.451208    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.451404 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.451251    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjwtz\" (UniqueName: \"kubernetes.io/projected/fe107d75-925a-4bf8-99db-c497e80dd4b1-kube-api-access-jjwtz\") pod \"maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.451404 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.451295    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-metrics-certs\") pod \"maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.452021 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.451988    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe107d75-925a-4bf8-99db-c497e80dd4b1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.454380 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.454347    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-metrics-certs\") pod \"maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.455255 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.455233    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.472752 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.472719    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjwtz\" (UniqueName: \"kubernetes.io/projected/fe107d75-925a-4bf8-99db-c497e80dd4b1-kube-api-access-jjwtz\") pod \"maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.587364 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.587280    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:00.927824 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:00.927796    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"]
Oct 30 00:27:00.929305 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:27:00.929277    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe107d75_925a_4bf8_99db_c497e80dd4b1.slice/crio-66f3c9663102d149769277bfbe57cf2c474f3701d6b7d8cb293939a52413658d WatchSource:0}: Error finding container 66f3c9663102d149769277bfbe57cf2c474f3701d6b7d8cb293939a52413658d: Status 404 returned error can't find the container with id 66f3c9663102d149769277bfbe57cf2c474f3701d6b7d8cb293939a52413658d
Oct 30 00:27:01.018614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:01.018578    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87" event={"ID":"fe107d75-925a-4bf8-99db-c497e80dd4b1","Type":"ContainerStarted","Data":"66f3c9663102d149769277bfbe57cf2c474f3701d6b7d8cb293939a52413658d"}
Oct 30 00:27:02.029187 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:02.029152    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87" event={"ID":"fe107d75-925a-4bf8-99db-c497e80dd4b1","Type":"ContainerStarted","Data":"0fc40917dc66aae9eabd7f0cda5024e9aee4bb82372ecf21ef395c73292502d5"}
Oct 30 00:27:02.029588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:02.029223    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:02.068941 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:02.068891    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87" podStartSLOduration=1.694707351 podStartE2EDuration="2.068877262s" totalImagesPullingTime="374.169911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:02.065990551 +0000 UTC m=+1331.591284430" watchObservedRunningTime="2026-10-30 00:27:02.068877262 +0000 UTC m=+1331.594171164"
Oct 30 00:27:03.915735 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:27:03.915692    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:27:04.275570 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:04.275538    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"]
Oct 30 00:27:04.319706 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:04.319675    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"]
Oct 30 00:27:04.320072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:04.319971    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"
Oct 30 00:27:04.384777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:04.384735    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-43dc72-dockercfg-j8dvs\""
Oct 30 00:27:04.385054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:04.385018    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-43dc72\""
Oct 30 00:27:04.427756 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:04.417193    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768f98d2-4192-4df7-8b73-19c8b39029b4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-43dc72-db8996694-scsv4\" (UID: \"768f98d2-4192-4df7-8b73-19c8b39029b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"
Oct 30 00:27:04.427756 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:04.417434    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k587\" (UniqueName: \"kubernetes.io/projected/768f98d2-4192-4df7-8b73-19c8b39029b4-kube-api-access-6k587\") pod \"payload-processing-e2e-api-b-43dc72-db8996694-scsv4\" (UID: \"768f98d2-4192-4df7-8b73-19c8b39029b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"
Oct 30 00:27:04.521213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:04.521160    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768f98d2-4192-4df7-8b73-19c8b39029b4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-43dc72-db8996694-scsv4\" (UID: \"768f98d2-4192-4df7-8b73-19c8b39029b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"
Oct 30 00:27:04.521424 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:04.521386    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k587\" (UniqueName: \"kubernetes.io/projected/768f98d2-4192-4df7-8b73-19c8b39029b4-kube-api-access-6k587\") pod \"payload-processing-e2e-api-b-43dc72-db8996694-scsv4\" (UID: \"768f98d2-4192-4df7-8b73-19c8b39029b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"
Oct 30 00:27:04.522667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:04.522640    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768f98d2-4192-4df7-8b73-19c8b39029b4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-43dc72-db8996694-scsv4\" (UID: \"768f98d2-4192-4df7-8b73-19c8b39029b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"
Oct 30 00:27:04.546257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:04.545180    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k587\" (UniqueName: \"kubernetes.io/projected/768f98d2-4192-4df7-8b73-19c8b39029b4-kube-api-access-6k587\") pod \"payload-processing-e2e-api-b-43dc72-db8996694-scsv4\" (UID: \"768f98d2-4192-4df7-8b73-19c8b39029b4\") " pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"
Oct 30 00:27:04.688631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:04.688598    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"
Oct 30 00:27:05.007451 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:05.007417    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"]
Oct 30 00:27:05.007891 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:27:05.007862    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768f98d2_4192_4df7_8b73_19c8b39029b4.slice/crio-10772de9546509f287e3ec89a4a59f04a09fe9385b6426210cecedc0b78ceb7c WatchSource:0}: Error finding container 10772de9546509f287e3ec89a4a59f04a09fe9385b6426210cecedc0b78ceb7c: Status 404 returned error can't find the container with id 10772de9546509f287e3ec89a4a59f04a09fe9385b6426210cecedc0b78ceb7c
Oct 30 00:27:05.051878 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:05.051844    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4" event={"ID":"768f98d2-4192-4df7-8b73-19c8b39029b4","Type":"ContainerStarted","Data":"10772de9546509f287e3ec89a4a59f04a09fe9385b6426210cecedc0b78ceb7c"}
Oct 30 00:27:06.060429 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:06.060396    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4" event={"ID":"768f98d2-4192-4df7-8b73-19c8b39029b4","Type":"ContainerStarted","Data":"2783553b201e9f8fe3a5126f367fa5a8edf50a051c67663a35342786ee959f43"}
Oct 30 00:27:06.060812 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:06.060530    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"
Oct 30 00:27:06.098762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:06.098711    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4" podStartSLOduration=2.098694683 podStartE2EDuration="2.098694683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:06.095535678 +0000 UTC m=+1335.620829557" watchObservedRunningTime="2026-10-30 00:27:06.098694683 +0000 UTC m=+1335.623988560"
Oct 30 00:27:08.043533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:08.043505    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:27:10.525329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.525288    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"]
Oct 30 00:27:10.532017 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.531988    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"
Oct 30 00:27:10.549449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.549412    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"]
Oct 30 00:27:10.552530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.552509    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-00df5f\""
Oct 30 00:27:10.584565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.584535    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"]
Oct 30 00:27:10.588880 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.588861    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"
Oct 30 00:27:10.605043 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.605012    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-00df5f-dockercfg-fb77c\""
Oct 30 00:27:10.615674 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.615629    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"]
Oct 30 00:27:10.769093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.766323    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxx4\" (UniqueName: \"kubernetes.io/projected/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-kube-api-access-xsxx4\") pod \"payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb\" (UID: \"71a1c040-83ea-4f04-b8a3-e3f3c59937d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"
Oct 30 00:27:10.769093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.766582    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f51de93-61c4-40c5-808f-347d21189d3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n\" (UID: \"1f51de93-61c4-40c5-808f-347d21189d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"
Oct 30 00:27:10.769093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.766733    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb\" (UID: \"71a1c040-83ea-4f04-b8a3-e3f3c59937d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"
Oct 30 00:27:10.769093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.766830    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxvpp\" (UniqueName: \"kubernetes.io/projected/1f51de93-61c4-40c5-808f-347d21189d3d-kube-api-access-pxvpp\") pod \"payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n\" (UID: \"1f51de93-61c4-40c5-808f-347d21189d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"
Oct 30 00:27:10.868423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.868323    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb\" (UID: \"71a1c040-83ea-4f04-b8a3-e3f3c59937d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"
Oct 30 00:27:10.868760 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.868733    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvpp\" (UniqueName: \"kubernetes.io/projected/1f51de93-61c4-40c5-808f-347d21189d3d-kube-api-access-pxvpp\") pod \"payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n\" (UID: \"1f51de93-61c4-40c5-808f-347d21189d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"
Oct 30 00:27:10.868899 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.868804    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxx4\" (UniqueName: \"kubernetes.io/projected/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-kube-api-access-xsxx4\") pod \"payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb\" (UID: \"71a1c040-83ea-4f04-b8a3-e3f3c59937d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"
Oct 30 00:27:10.869069 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.869014    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f51de93-61c4-40c5-808f-347d21189d3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n\" (UID: \"1f51de93-61c4-40c5-808f-347d21189d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"
Oct 30 00:27:10.869314 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.869287    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb\" (UID: \"71a1c040-83ea-4f04-b8a3-e3f3c59937d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"
Oct 30 00:27:10.869966 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.869941    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f51de93-61c4-40c5-808f-347d21189d3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n\" (UID: \"1f51de93-61c4-40c5-808f-347d21189d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"
Oct 30 00:27:10.890641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.890616    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxx4\" (UniqueName: \"kubernetes.io/projected/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-kube-api-access-xsxx4\") pod \"payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb\" (UID: \"71a1c040-83ea-4f04-b8a3-e3f3c59937d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"
Oct 30 00:27:10.891131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.891109    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvpp\" (UniqueName: \"kubernetes.io/projected/1f51de93-61c4-40c5-808f-347d21189d3d-kube-api-access-pxvpp\") pod \"payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n\" (UID: \"1f51de93-61c4-40c5-808f-347d21189d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"
Oct 30 00:27:10.906495 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:10.906461    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"
Oct 30 00:27:11.151838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:11.151753    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"
Oct 30 00:27:11.284584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:11.284550    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"]
Oct 30 00:27:11.286483 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:27:11.286452    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f51de93_61c4_40c5_808f_347d21189d3d.slice/crio-49b7217f8e03b4cd3a7dc5ee5c10146e0be03e1a75b2caadfe899ea60a2904be WatchSource:0}: Error finding container 49b7217f8e03b4cd3a7dc5ee5c10146e0be03e1a75b2caadfe899ea60a2904be: Status 404 returned error can't find the container with id 49b7217f8e03b4cd3a7dc5ee5c10146e0be03e1a75b2caadfe899ea60a2904be
Oct 30 00:27:11.462940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:11.462910    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"]
Oct 30 00:27:11.465005 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:27:11.464975    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a1c040_83ea_4f04_b8a3_e3f3c59937d9.slice/crio-7f41f38d02fa35344b5d68339a72af49f85b814edeaa13b1aaca5e1832cb5048 WatchSource:0}: Error finding container 7f41f38d02fa35344b5d68339a72af49f85b814edeaa13b1aaca5e1832cb5048: Status 404 returned error can't find the container with id 7f41f38d02fa35344b5d68339a72af49f85b814edeaa13b1aaca5e1832cb5048
Oct 30 00:27:12.120098 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:12.120057    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n" event={"ID":"1f51de93-61c4-40c5-808f-347d21189d3d","Type":"ContainerStarted","Data":"095bdfefc4ca838c71449bd37ca8f95fce0fed63a95986534e75655865e1530a"}
Oct 30 00:27:12.120609 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:12.120128    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n" event={"ID":"1f51de93-61c4-40c5-808f-347d21189d3d","Type":"ContainerStarted","Data":"49b7217f8e03b4cd3a7dc5ee5c10146e0be03e1a75b2caadfe899ea60a2904be"}
Oct 30 00:27:12.120609 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:12.120226    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"
Oct 30 00:27:12.121931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:12.121904    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb" event={"ID":"71a1c040-83ea-4f04-b8a3-e3f3c59937d9","Type":"ContainerStarted","Data":"fe745c27e24224794999671dd85362f3d51814c8cd024e6f7e335c71e5b99d18"}
Oct 30 00:27:12.122087 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:12.121952    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb" event={"ID":"71a1c040-83ea-4f04-b8a3-e3f3c59937d9","Type":"ContainerStarted","Data":"7f41f38d02fa35344b5d68339a72af49f85b814edeaa13b1aaca5e1832cb5048"}
Oct 30 00:27:12.122087 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:12.121985    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"
Oct 30 00:27:12.160285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:12.160228    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n" podStartSLOduration=2.160209404 podStartE2EDuration="2.160209404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:12.154726037 +0000 UTC m=+1341.680019915" watchObservedRunningTime="2026-10-30 00:27:12.160209404 +0000 UTC m=+1341.685503281"
Oct 30 00:27:12.193695 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:12.193643    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb" podStartSLOduration=2.193629123 podStartE2EDuration="2.193629123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:12.190004867 +0000 UTC m=+1341.715298750" watchObservedRunningTime="2026-10-30 00:27:12.193629123 +0000 UTC m=+1341.718922999"
Oct 30 00:27:14.887558 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:14.887519    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-877hw"
Oct 30 00:27:15.581324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:15.581287    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"]
Oct 30 00:27:15.581529 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:15.581476    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" containerID="cri-o://10b4317c4a0f9e5d0367b20f935ffce503213edd90ae0bb6b97e18b7555cf233" gracePeriod=30
Oct 30 00:27:15.701932 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:15.701569    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:27:15.701932 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:15.701653    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:27:17.071734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:17.071705    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"
Oct 30 00:27:19.576478 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:19.576439    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:27:21.004306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.000422    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:27:21.090413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.087979    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-credential-socket\") pod \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") "
Oct 30 00:27:21.090413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.088125    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-token\" (UniqueName: \"kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-token\") pod \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") "
Oct 30 00:27:21.090413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.088201    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-kube-api-access-n82rd\" (UniqueName: \"kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-kube-api-access-n82rd\") pod \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") "
Oct 30 00:27:21.090413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.088293    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-data\") pod \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") "
Oct 30 00:27:21.090413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.088419    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-envoy\") pod \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") "
Oct 30 00:27:21.090413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.088484    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-socket\") pod \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") "
Oct 30 00:27:21.090413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.088534    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istiod-ca-cert\") pod \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") "
Oct 30 00:27:21.090413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.088586    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-podinfo\") pod \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") "
Oct 30 00:27:21.090413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.088630    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-certs\") pod \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\" (UID: \"a5e3960f-82b3-4e2d-af34-44b4d6b3abec\") "
Oct 30 00:27:21.100342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.091954    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-data" pod "a5e3960f-82b3-4e2d-af34-44b4d6b3abec" (UID: "a5e3960f-82b3-4e2d-af34-44b4d6b3abec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:27:21.100342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.092303    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-credential-socket" pod "a5e3960f-82b3-4e2d-af34-44b4d6b3abec" (UID: "a5e3960f-82b3-4e2d-af34-44b4d6b3abec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:27:21.100342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.092398    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-socket" pod "a5e3960f-82b3-4e2d-af34-44b4d6b3abec" (UID: "a5e3960f-82b3-4e2d-af34-44b4d6b3abec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:27:21.100342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.093108    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-certs" pod "a5e3960f-82b3-4e2d-af34-44b4d6b3abec" (UID: "a5e3960f-82b3-4e2d-af34-44b4d6b3abec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:27:21.100342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.097777    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istiod-ca-cert" pod "a5e3960f-82b3-4e2d-af34-44b4d6b3abec" (UID: "a5e3960f-82b3-4e2d-af34-44b4d6b3abec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:27:21.101845 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.101443    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-token" pod "a5e3960f-82b3-4e2d-af34-44b4d6b3abec" (UID: "a5e3960f-82b3-4e2d-af34-44b4d6b3abec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:27:21.104157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.104004    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-envoy" pod "a5e3960f-82b3-4e2d-af34-44b4d6b3abec" (UID: "a5e3960f-82b3-4e2d-af34-44b4d6b3abec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:27:21.120634 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.120589    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-kube-api-access-n82rd" pod "a5e3960f-82b3-4e2d-af34-44b4d6b3abec" (UID: "a5e3960f-82b3-4e2d-af34-44b4d6b3abec"). InnerVolumeSpecName "kube-api-access-n82rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:27:21.121157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.121126    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-podinfo" pod "a5e3960f-82b3-4e2d-af34-44b4d6b3abec" (UID: "a5e3960f-82b3-4e2d-af34-44b4d6b3abec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:27:21.201068 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.201020    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:21.201068 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.201066    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:21.201362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.201090    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:21.201362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.201109    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:21.201362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.201117    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:21.201362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.201126    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:21.201362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.201136    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n82rd\" (UniqueName: \"kubernetes.io/projected/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-kube-api-access-n82rd\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:21.201362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.201145    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:21.201362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.201153    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e3960f-82b3-4e2d-af34-44b4d6b3abec-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:21.229073 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.229038    4441 generic.go:356] "Generic (PLEG): container finished" podID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerID="10b4317c4a0f9e5d0367b20f935ffce503213edd90ae0bb6b97e18b7555cf233" exitCode=0
Oct 30 00:27:21.229286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.229121    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"
Oct 30 00:27:21.229374 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.229320    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" event={"ID":"a5e3960f-82b3-4e2d-af34-44b4d6b3abec","Type":"ContainerDied","Data":"10b4317c4a0f9e5d0367b20f935ffce503213edd90ae0bb6b97e18b7555cf233"}
Oct 30 00:27:21.229456 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.229379    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl" event={"ID":"a5e3960f-82b3-4e2d-af34-44b4d6b3abec","Type":"ContainerDied","Data":"96caa3dc224b624e9191379161852d946152299b75f7c17cc0f3c4ad8d71d729"}
Oct 30 00:27:21.229456 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.229423    4441 scope.go:122] "RemoveContainer" containerID="10b4317c4a0f9e5d0367b20f935ffce503213edd90ae0bb6b97e18b7555cf233"
Oct 30 00:27:21.260384 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.260350    4441 scope.go:122] "RemoveContainer" containerID="10b4317c4a0f9e5d0367b20f935ffce503213edd90ae0bb6b97e18b7555cf233"
Oct 30 00:27:21.260836 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:27:21.260789    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b4317c4a0f9e5d0367b20f935ffce503213edd90ae0bb6b97e18b7555cf233\": container with ID starting with 10b4317c4a0f9e5d0367b20f935ffce503213edd90ae0bb6b97e18b7555cf233 not found: ID does not exist" containerID="10b4317c4a0f9e5d0367b20f935ffce503213edd90ae0bb6b97e18b7555cf233"
Oct 30 00:27:21.260950 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.260842    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b4317c4a0f9e5d0367b20f935ffce503213edd90ae0bb6b97e18b7555cf233"} err="failed to get container status \"10b4317c4a0f9e5d0367b20f935ffce503213edd90ae0bb6b97e18b7555cf233\": rpc error: code = NotFound desc = could not find container \"10b4317c4a0f9e5d0367b20f935ffce503213edd90ae0bb6b97e18b7555cf233\": container with ID starting with 10b4317c4a0f9e5d0367b20f935ffce503213edd90ae0bb6b97e18b7555cf233 not found: ID does not exist"
Oct 30 00:27:21.356810 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.356772    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"]
Oct 30 00:27:21.365414 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:21.365384    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-r7twl"]
Oct 30 00:27:22.677144 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:22.676614    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" path="/var/lib/kubelet/pods/a5e3960f-82b3-4e2d-af34-44b4d6b3abec/volumes"
Oct 30 00:27:22.906047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:22.905953    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"]
Oct 30 00:27:22.958583 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:22.958482    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"]
Oct 30 00:27:23.156005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:23.155977    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"
Oct 30 00:27:23.157818 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:23.156593    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"
Oct 30 00:27:27.906697 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:27.906658    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f" podUID="af5d056f-dec9-4092-999d-60b735e96e06" containerName="payload-processing" containerID="cri-o://a4066e887ee99a3e82988b505af53db4b0797d2632301bc2b61603c30a991f5d" gracePeriod=25
Oct 30 00:27:27.959157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:27.959111    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft" podUID="4906b857-b547-40b4-bf72-8829b532e404" containerName="payload-pre-processing" containerID="cri-o://325e06ea497a7b1dfaa07d7d1a88d686f13644321ae0b01ecec8d796b1a3c029" gracePeriod=25
Oct 30 00:27:28.423350 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.421513    4441 generic.go:356] "Generic (PLEG): container finished" podID="af5d056f-dec9-4092-999d-60b735e96e06" containerID="a4066e887ee99a3e82988b505af53db4b0797d2632301bc2b61603c30a991f5d" exitCode=0
Oct 30 00:27:28.423350 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.421574    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f" event={"ID":"af5d056f-dec9-4092-999d-60b735e96e06","Type":"ContainerDied","Data":"a4066e887ee99a3e82988b505af53db4b0797d2632301bc2b61603c30a991f5d"}
Oct 30 00:27:28.423350 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.421667    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f" event={"ID":"af5d056f-dec9-4092-999d-60b735e96e06","Type":"ContainerDied","Data":"1f3144b419f70aca72cb266de4fff73e909533b31c609c29d99670db851402eb"}
Oct 30 00:27:28.423350 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.421692    4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f3144b419f70aca72cb266de4fff73e909533b31c609c29d99670db851402eb"
Oct 30 00:27:28.425976 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.425950    4441 generic.go:356] "Generic (PLEG): container finished" podID="4906b857-b547-40b4-bf72-8829b532e404" containerID="325e06ea497a7b1dfaa07d7d1a88d686f13644321ae0b01ecec8d796b1a3c029" exitCode=0
Oct 30 00:27:28.426160 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.426086    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft" event={"ID":"4906b857-b547-40b4-bf72-8829b532e404","Type":"ContainerDied","Data":"325e06ea497a7b1dfaa07d7d1a88d686f13644321ae0b01ecec8d796b1a3c029"}
Oct 30 00:27:28.446423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.446394    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"
Oct 30 00:27:28.455257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.455229    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"
Oct 30 00:27:28.522140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.521701    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5d056f-dec9-4092-999d-60b735e96e06-kube-api-access-p95qt\" (UniqueName: \"kubernetes.io/projected/af5d056f-dec9-4092-999d-60b735e96e06-kube-api-access-p95qt\") pod \"af5d056f-dec9-4092-999d-60b735e96e06\" (UID: \"af5d056f-dec9-4092-999d-60b735e96e06\") "
Oct 30 00:27:28.522140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.521753    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4906b857-b547-40b4-bf72-8829b532e404-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4906b857-b547-40b4-bf72-8829b532e404-plugins-config-volume\") pod \"4906b857-b547-40b4-bf72-8829b532e404\" (UID: \"4906b857-b547-40b4-bf72-8829b532e404\") "
Oct 30 00:27:28.522140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.521967    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4906b857-b547-40b4-bf72-8829b532e404-kube-api-access-msc2x\" (UniqueName: \"kubernetes.io/projected/4906b857-b547-40b4-bf72-8829b532e404-kube-api-access-msc2x\") pod \"4906b857-b547-40b4-bf72-8829b532e404\" (UID: \"4906b857-b547-40b4-bf72-8829b532e404\") "
Oct 30 00:27:28.522140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.522103    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af5d056f-dec9-4092-999d-60b735e96e06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5d056f-dec9-4092-999d-60b735e96e06-plugins-config-volume\") pod \"af5d056f-dec9-4092-999d-60b735e96e06\" (UID: \"af5d056f-dec9-4092-999d-60b735e96e06\") "
Oct 30 00:27:28.523136 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.523089    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4906b857-b547-40b4-bf72-8829b532e404-plugins-config-volume" pod "4906b857-b547-40b4-bf72-8829b532e404" (UID: "4906b857-b547-40b4-bf72-8829b532e404"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:27:28.523136 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.523103    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5d056f-dec9-4092-999d-60b735e96e06-plugins-config-volume" pod "af5d056f-dec9-4092-999d-60b735e96e06" (UID: "af5d056f-dec9-4092-999d-60b735e96e06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:27:28.529539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.526934    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5d056f-dec9-4092-999d-60b735e96e06-kube-api-access-p95qt" pod "af5d056f-dec9-4092-999d-60b735e96e06" (UID: "af5d056f-dec9-4092-999d-60b735e96e06"). InnerVolumeSpecName "kube-api-access-p95qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:27:28.529539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.527686    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4906b857-b547-40b4-bf72-8829b532e404-kube-api-access-msc2x" pod "4906b857-b547-40b4-bf72-8829b532e404" (UID: "4906b857-b547-40b4-bf72-8829b532e404"). InnerVolumeSpecName "kube-api-access-msc2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:27:28.625156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.625126    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msc2x\" (UniqueName: \"kubernetes.io/projected/4906b857-b547-40b4-bf72-8829b532e404-kube-api-access-msc2x\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:28.625156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.625156    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5d056f-dec9-4092-999d-60b735e96e06-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:28.625418 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.625172    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p95qt\" (UniqueName: \"kubernetes.io/projected/af5d056f-dec9-4092-999d-60b735e96e06-kube-api-access-p95qt\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:28.625418 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:28.625185    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4906b857-b547-40b4-bf72-8829b532e404-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:29.436106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:29.436076    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"
Oct 30 00:27:29.436616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:29.436072    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft" event={"ID":"4906b857-b547-40b4-bf72-8829b532e404","Type":"ContainerDied","Data":"86bea419a5b90ddd2483562dc37d3d4d0c991829a1b4f025ff6c2852eb24bcdf"}
Oct 30 00:27:29.436616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:29.436227    4441 scope.go:122] "RemoveContainer" containerID="325e06ea497a7b1dfaa07d7d1a88d686f13644321ae0b01ecec8d796b1a3c029"
Oct 30 00:27:29.436616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:29.436462    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"
Oct 30 00:27:29.483176 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:29.483152    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"]
Oct 30 00:27:29.488576 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:29.488548    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f649b0-7d747755d8-gq92f"]
Oct 30 00:27:29.523320 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:29.523284    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"]
Oct 30 00:27:29.528757 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:29.528722    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-66479f76f9-jk9ft"]
Oct 30 00:27:30.612229 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:30.612199    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4906b857-b547-40b4-bf72-8829b532e404" path="/var/lib/kubelet/pods/4906b857-b547-40b4-bf72-8829b532e404/volumes"
Oct 30 00:27:30.612755 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:30.612739    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af5d056f-dec9-4092-999d-60b735e96e06" path="/var/lib/kubelet/pods/af5d056f-dec9-4092-999d-60b735e96e06/volumes"
Oct 30 00:27:31.166418 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.166384    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"]
Oct 30 00:27:31.167526 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.167503    4441 cpu_manager.go:414] "containerMap: removing container" podUID="4906b857-b547-40b4-bf72-8829b532e404" containerName="payload-pre-processing"
Oct 30 00:27:31.167618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.167531    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4906b857-b547-40b4-bf72-8829b532e404" containerName="payload-pre-processing"
Oct 30 00:27:31.167618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.167559    4441 cpu_manager.go:414] "containerMap: removing container" podUID="af5d056f-dec9-4092-999d-60b735e96e06" containerName="payload-processing"
Oct 30 00:27:31.167618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.167568    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af5d056f-dec9-4092-999d-60b735e96e06" containerName="payload-processing"
Oct 30 00:27:31.167618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.167598    4441 cpu_manager.go:414] "containerMap: removing container" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy"
Oct 30 00:27:31.167618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.167606    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy"
Oct 30 00:27:31.167822 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.167809    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4906b857-b547-40b4-bf72-8829b532e404" containerName="payload-pre-processing"
Oct 30 00:27:31.167858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.167828    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="af5d056f-dec9-4092-999d-60b735e96e06" containerName="payload-processing"
Oct 30 00:27:31.167858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.167846    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e3960f-82b3-4e2d-af34-44b4d6b3abec" containerName="istio-proxy"
Oct 30 00:27:31.183660 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.180772    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:31.195597 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.193564    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"]
Oct 30 00:27:31.197312 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.195828    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-43dc72-serving-cert\""
Oct 30 00:27:31.292945 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.292899    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k75st\" (UniqueName: \"kubernetes.io/projected/18454967-0713-4eb4-93a6-7806304ea164-kube-api-access-k75st\") pod \"maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:31.293206 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.293178    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-maas-api-tls\") pod \"maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:31.293295 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.293284    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-metrics-certs\") pod \"maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:31.293361 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.293327    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18454967-0713-4eb4-93a6-7806304ea164-openshift-service-ca\") pod \"maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:31.395295 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.395258    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-maas-api-tls\") pod \"maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:31.395510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.395379    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-metrics-certs\") pod \"maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:31.395510 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:27:31.395407    4441 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-serving-cert: secret "maas-api-e2e-api-b-43dc72-serving-cert" not found
Oct 30 00:27:31.395510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.395420    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18454967-0713-4eb4-93a6-7806304ea164-openshift-service-ca\") pod \"maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:31.395510 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:27:31.395485    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-maas-api-tls podName:18454967-0713-4eb4-93a6-7806304ea164 nodeName:}" failed. No retries permitted until 2026-10-30 00:27:31.895461853 +0000 UTC m=+1361.420755711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-maas-api-tls") pod "maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td" (UID: "18454967-0713-4eb4-93a6-7806304ea164") : secret "maas-api-e2e-api-b-43dc72-serving-cert" not found
Oct 30 00:27:31.395800 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.395549    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k75st\" (UniqueName: \"kubernetes.io/projected/18454967-0713-4eb4-93a6-7806304ea164-kube-api-access-k75st\") pod \"maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:31.396294 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.396267    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18454967-0713-4eb4-93a6-7806304ea164-openshift-service-ca\") pod \"maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:31.398732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.398706    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-metrics-certs\") pod \"maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:31.418972 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.418910    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k75st\" (UniqueName: \"kubernetes.io/projected/18454967-0713-4eb4-93a6-7806304ea164-kube-api-access-k75st\") pod \"maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:31.904125 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.902899    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-maas-api-tls\") pod \"maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:31.918863 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:31.918828    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-maas-api-tls\") pod \"maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:32.141871 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:32.141829    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:32.531253 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:32.531219    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"]
Oct 30 00:27:32.532097 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:27:32.532068    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18454967_0713_4eb4_93a6_7806304ea164.slice/crio-0de2882ec04fe995706e6fc874ab940f0816875008b675578e5ae436e93d025d WatchSource:0}: Error finding container 0de2882ec04fe995706e6fc874ab940f0816875008b675578e5ae436e93d025d: Status 404 returned error can't find the container with id 0de2882ec04fe995706e6fc874ab940f0816875008b675578e5ae436e93d025d
Oct 30 00:27:32.865191 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:32.865108    4441 patch_prober.go:29] interesting pod/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:27:32.933481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:32.865353    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:27:33.477099 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.472378    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"]
Oct 30 00:27:33.527704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.510019    4441 generic.go:356] "Generic (PLEG): container finished" podID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerID="58987af5c64e3fa80dd4753614ce45bfded62887cdb075f2e662b7b95bf94211" exitCode=0
Oct 30 00:27:33.527704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.510206    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td" event={"ID":"18454967-0713-4eb4-93a6-7806304ea164","Type":"ContainerStarted","Data":"60348bdfe9df09a69d685179992bf240664ab0f9e5b64205840644f1f0cfb1a8"}
Oct 30 00:27:33.527704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.510301    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"
Oct 30 00:27:33.527704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.511837    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td" event={"ID":"18454967-0713-4eb4-93a6-7806304ea164","Type":"ContainerStarted","Data":"0de2882ec04fe995706e6fc874ab940f0816875008b675578e5ae436e93d025d"}
Oct 30 00:27:33.527704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.511875    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" event={"ID":"4a972f67-aa0b-4444-9b0a-9662f67e6cac","Type":"ContainerDied","Data":"58987af5c64e3fa80dd4753614ce45bfded62887cdb075f2e662b7b95bf94211"}
Oct 30 00:27:33.527704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.511908    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:33.527704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.512365    4441 scope.go:122] "RemoveContainer" containerID="58987af5c64e3fa80dd4753614ce45bfded62887cdb075f2e662b7b95bf94211"
Oct 30 00:27:33.527704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.519542    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:27:33.527704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.524060    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d3e13d-dockercfg-6s9vl\""
Oct 30 00:27:33.527704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.524138    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d3e13d\""
Oct 30 00:27:33.531400 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.531379    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"]
Oct 30 00:27:33.533657 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.533492    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:27:33.533657 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.533604    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:27:33.598063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.597945    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td" podStartSLOduration=2.136701742 podStartE2EDuration="2.597922524s" totalImagesPullingTime="461.220782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:33.552729281 +0000 UTC m=+1363.078023159" watchObservedRunningTime="2026-10-30 00:27:33.597922524 +0000 UTC m=+1363.123216401"
Oct 30 00:27:33.681156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.681114    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvzc7\" (UniqueName: \"kubernetes.io/projected/f80a0aed-06cb-4ca3-a9e4-142a932076fe-kube-api-access-xvzc7\") pod \"payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4\" (UID: \"f80a0aed-06cb-4ca3-a9e4-142a932076fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"
Oct 30 00:27:33.681338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.681285    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80a0aed-06cb-4ca3-a9e4-142a932076fe-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4\" (UID: \"f80a0aed-06cb-4ca3-a9e4-142a932076fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"
Oct 30 00:27:33.785479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.783685    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvzc7\" (UniqueName: \"kubernetes.io/projected/f80a0aed-06cb-4ca3-a9e4-142a932076fe-kube-api-access-xvzc7\") pod \"payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4\" (UID: \"f80a0aed-06cb-4ca3-a9e4-142a932076fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"
Oct 30 00:27:33.785479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.783813    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80a0aed-06cb-4ca3-a9e4-142a932076fe-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4\" (UID: \"f80a0aed-06cb-4ca3-a9e4-142a932076fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"
Oct 30 00:27:33.785479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.785392    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80a0aed-06cb-4ca3-a9e4-142a932076fe-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4\" (UID: \"f80a0aed-06cb-4ca3-a9e4-142a932076fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"
Oct 30 00:27:33.805939 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.805905    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvzc7\" (UniqueName: \"kubernetes.io/projected/f80a0aed-06cb-4ca3-a9e4-142a932076fe-kube-api-access-xvzc7\") pod \"payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4\" (UID: \"f80a0aed-06cb-4ca3-a9e4-142a932076fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"
Oct 30 00:27:33.842362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:33.842097    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"
Oct 30 00:27:33.920636 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:27:33.920602    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:27:34.185901 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:34.185870    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"]
Oct 30 00:27:34.186317 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:27:34.185922    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80a0aed_06cb_4ca3_a9e4_142a932076fe.slice/crio-50c688ccf4a5ada0bf5bb552d2d25d5e4c62b74bf8c913d59dac05045ac65a82 WatchSource:0}: Error finding container 50c688ccf4a5ada0bf5bb552d2d25d5e4c62b74bf8c913d59dac05045ac65a82: Status 404 returned error can't find the container with id 50c688ccf4a5ada0bf5bb552d2d25d5e4c62b74bf8c913d59dac05045ac65a82
Oct 30 00:27:34.519549 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:34.519515    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" event={"ID":"4a972f67-aa0b-4444-9b0a-9662f67e6cac","Type":"ContainerStarted","Data":"a23b8be31f268771894461bd59f3a641792f43a2489b694846ef35d6a380c23d"}
Oct 30 00:27:34.524067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:34.522501    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4" event={"ID":"f80a0aed-06cb-4ca3-a9e4-142a932076fe","Type":"ContainerStarted","Data":"9891cb39a655643f9f371e44b32372531e010f4d77d06d10ea95619ddaf891fe"}
Oct 30 00:27:34.524067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:34.522553    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4" event={"ID":"f80a0aed-06cb-4ca3-a9e4-142a932076fe","Type":"ContainerStarted","Data":"50c688ccf4a5ada0bf5bb552d2d25d5e4c62b74bf8c913d59dac05045ac65a82"}
Oct 30 00:27:34.524067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:34.522580    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"
Oct 30 00:27:34.609518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:34.609467    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4" podStartSLOduration=1.609452924 podStartE2EDuration="1.609452924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:34.603442223 +0000 UTC m=+1364.128736100" watchObservedRunningTime="2026-10-30 00:27:34.609452924 +0000 UTC m=+1364.134746801"
Oct 30 00:27:34.744046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:34.743454    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:27:34.744046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:34.743589    4441 patch_prober.go:29] interesting pod/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 00:27:34.744046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:34.743636    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 00:27:35.746049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:35.745455    4441 patch_prober.go:29] interesting pod/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 00:27:35.746049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:35.745520    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 00:27:35.890393 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:35.890357    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"]
Oct 30 00:27:35.894391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:35.894364    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:35.908961 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:35.908936    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6cfcf7-openshift-default-dockercfg-vxfhf\""
Oct 30 00:27:35.987759 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:35.987725    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"]
Oct 30 00:27:36.084173 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.084084    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"]
Oct 30 00:27:36.085841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.085810    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-token\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.085952 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.085908    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-podinfo\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.085994 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.085975    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-data\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.086086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.086069    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-certs\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.086211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.086189    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5b0544-84cb-494c-a40d-7b23774cc079-istiod-ca-cert\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.086255 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.086236    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-credential-socket\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.086290 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.086259    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-socket\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.086342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.086332    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtlxs\" (UniqueName: \"kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-kube-api-access-gtlxs\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.086387 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.086376    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-envoy\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.161111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.161076    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"]
Oct 30 00:27:36.192161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.192125    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-certs\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.192471 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.192451    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5b0544-84cb-494c-a40d-7b23774cc079-istiod-ca-cert\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.192612 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.192599    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-credential-socket\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.192728 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.192714    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-socket\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.192918 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.192904    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtlxs\" (UniqueName: \"kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-kube-api-access-gtlxs\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.193122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.193103    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-envoy\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.193299 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.193285    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-token\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.193471 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.193457    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-podinfo\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.193619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.193605    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-data\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.193962 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.193937    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-socket\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.194146 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.194060    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-data\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.194525 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.194503    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-certs\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.194905 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.194883    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-credential-socket\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.195266 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.195081    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5b0544-84cb-494c-a40d-7b23774cc079-istiod-ca-cert\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.198587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.198561    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-podinfo\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.201803 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.200252    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-envoy\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.219046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.217900    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtlxs\" (UniqueName: \"kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-kube-api-access-gtlxs\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.219046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.218992    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-token\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.509036 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.508983    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:36.740965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.740538    4441 patch_prober.go:29] interesting pod/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 00:27:36.740965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.740596    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 00:27:36.986486 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.986452    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"]
Oct 30 00:27:36.988875 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:27:36.988843    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5b0544_84cb_494c_a40d_7b23774cc079.slice/crio-0eb03b94d306c067ae6e85685e268b02551f102a61aea489797522bdf7a17404 WatchSource:0}: Error finding container 0eb03b94d306c067ae6e85685e268b02551f102a61aea489797522bdf7a17404: Status 404 returned error can't find the container with id 0eb03b94d306c067ae6e85685e268b02551f102a61aea489797522bdf7a17404
Oct 30 00:27:36.992407 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.992360    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:27:36.992509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.992464    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:27:36.992588 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:36.992520    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:27:37.582082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:37.581191    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz" event={"ID":"9a5b0544-84cb-494c-a40d-7b23774cc079","Type":"ContainerStarted","Data":"fc686a206abcba3c8a4ed2b27ab1f5d51d8049ac1e79c16d3c94142461ba7fa7"}
Oct 30 00:27:37.582082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:37.581295    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz" event={"ID":"9a5b0544-84cb-494c-a40d-7b23774cc079","Type":"ContainerStarted","Data":"0eb03b94d306c067ae6e85685e268b02551f102a61aea489797522bdf7a17404"}
Oct 30 00:27:37.644050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:37.641552    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz" podStartSLOduration=2.641532911 podStartE2EDuration="2.641532911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:37.626149241 +0000 UTC m=+1367.151443119" watchObservedRunningTime="2026-10-30 00:27:37.641532911 +0000 UTC m=+1367.166826790"
Oct 30 00:27:37.747881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:37.747837    4441 patch_prober.go:29] interesting pod/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 00:27:37.748195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:37.748164    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 00:27:38.516050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:38.512170    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:38.516050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:38.512292    4441 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 00:27:38.516050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:38.512351    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz" podUID="9a5b0544-84cb-494c-a40d-7b23774cc079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 00:27:38.741293 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:38.741256    4441 patch_prober.go:29] interesting pod/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 00:27:38.741561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:38.741533    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 00:27:39.509664 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:39.509629    4441 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 00:27:39.509842 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:39.509695    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz" podUID="9a5b0544-84cb-494c-a40d-7b23774cc079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 00:27:39.580717 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:39.580683    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:27:39.810084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:39.803652    4441 patch_prober.go:29] interesting pod/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 00:27:39.810084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:39.803723    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 00:27:40.509439 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:40.509402    4441 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 00:27:40.509650 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:40.509458    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz" podUID="9a5b0544-84cb-494c-a40d-7b23774cc079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 00:27:40.757614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:40.757584    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:27:40.758378 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:40.758355    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:27:40.759487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:40.759432    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:27:41.085116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.084980    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv" podUID="cbd1c416-58d1-4259-aa8c-bbf63b887e3f" containerName="payload-processing" containerID="cri-o://3933e823187a61215d8e85ea5cbd89c045fe7d6f2e4a85dc480efd4ebbbf5ce4" gracePeriod=25
Oct 30 00:27:41.162373 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.162081    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7" podUID="a95c0bd0-aeea-40db-923b-acddedcac7c6" containerName="payload-pre-processing" containerID="cri-o://eeba9c7b5ea54a648c3634cde22fa508fedf7232abb1732c4122946f6aeeb5c8" gracePeriod=25
Oct 30 00:27:41.510605 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.509817    4441 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 00:27:41.510605 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.509863    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz" podUID="9a5b0544-84cb-494c-a40d-7b23774cc079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 00:27:41.618687 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.618216    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"
Oct 30 00:27:41.704171 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.704084    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-plugins-config-volume\") pod \"cbd1c416-58d1-4259-aa8c-bbf63b887e3f\" (UID: \"cbd1c416-58d1-4259-aa8c-bbf63b887e3f\") "
Oct 30 00:27:41.704360 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.704300    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-kube-api-access-jfz68\" (UniqueName: \"kubernetes.io/projected/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-kube-api-access-jfz68\") pod \"cbd1c416-58d1-4259-aa8c-bbf63b887e3f\" (UID: \"cbd1c416-58d1-4259-aa8c-bbf63b887e3f\") "
Oct 30 00:27:41.705290 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.705253    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-plugins-config-volume" pod "cbd1c416-58d1-4259-aa8c-bbf63b887e3f" (UID: "cbd1c416-58d1-4259-aa8c-bbf63b887e3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:27:41.706002 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.705922    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:41.710779 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.710740    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-kube-api-access-jfz68" pod "cbd1c416-58d1-4259-aa8c-bbf63b887e3f" (UID: "cbd1c416-58d1-4259-aa8c-bbf63b887e3f"). InnerVolumeSpecName "kube-api-access-jfz68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:27:41.751101 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.748767    4441 generic.go:356] "Generic (PLEG): container finished" podID="cbd1c416-58d1-4259-aa8c-bbf63b887e3f" containerID="3933e823187a61215d8e85ea5cbd89c045fe7d6f2e4a85dc480efd4ebbbf5ce4" exitCode=0
Oct 30 00:27:41.751101 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.748930    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"
Oct 30 00:27:41.751101 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.749782    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"
Oct 30 00:27:41.751101 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.749872    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv" event={"ID":"cbd1c416-58d1-4259-aa8c-bbf63b887e3f","Type":"ContainerDied","Data":"3933e823187a61215d8e85ea5cbd89c045fe7d6f2e4a85dc480efd4ebbbf5ce4"}
Oct 30 00:27:41.751101 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.749922    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv" event={"ID":"cbd1c416-58d1-4259-aa8c-bbf63b887e3f","Type":"ContainerDied","Data":"08867c4657969ffa1718aea26cd081d9a17d0b7142b4ff9a12ff45a3dd905552"}
Oct 30 00:27:41.751101 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.749953    4441 scope.go:122] "RemoveContainer" containerID="3933e823187a61215d8e85ea5cbd89c045fe7d6f2e4a85dc480efd4ebbbf5ce4"
Oct 30 00:27:41.761856 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.760744    4441 generic.go:356] "Generic (PLEG): container finished" podID="a95c0bd0-aeea-40db-923b-acddedcac7c6" containerID="eeba9c7b5ea54a648c3634cde22fa508fedf7232abb1732c4122946f6aeeb5c8" exitCode=0
Oct 30 00:27:41.761856 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.761794    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7" event={"ID":"a95c0bd0-aeea-40db-923b-acddedcac7c6","Type":"ContainerDied","Data":"eeba9c7b5ea54a648c3634cde22fa508fedf7232abb1732c4122946f6aeeb5c8"}
Oct 30 00:27:41.776177 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.776157    4441 scope.go:122] "RemoveContainer" containerID="3933e823187a61215d8e85ea5cbd89c045fe7d6f2e4a85dc480efd4ebbbf5ce4"
Oct 30 00:27:41.776442 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:27:41.776415    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3933e823187a61215d8e85ea5cbd89c045fe7d6f2e4a85dc480efd4ebbbf5ce4\": container with ID starting with 3933e823187a61215d8e85ea5cbd89c045fe7d6f2e4a85dc480efd4ebbbf5ce4 not found: ID does not exist" containerID="3933e823187a61215d8e85ea5cbd89c045fe7d6f2e4a85dc480efd4ebbbf5ce4"
Oct 30 00:27:41.776497 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.776451    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3933e823187a61215d8e85ea5cbd89c045fe7d6f2e4a85dc480efd4ebbbf5ce4"} err="failed to get container status \"3933e823187a61215d8e85ea5cbd89c045fe7d6f2e4a85dc480efd4ebbbf5ce4\": rpc error: code = NotFound desc = could not find container \"3933e823187a61215d8e85ea5cbd89c045fe7d6f2e4a85dc480efd4ebbbf5ce4\": container with ID starting with 3933e823187a61215d8e85ea5cbd89c045fe7d6f2e4a85dc480efd4ebbbf5ce4 not found: ID does not exist"
Oct 30 00:27:41.776497 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.776467    4441 scope.go:122] "RemoveContainer" containerID="eeba9c7b5ea54a648c3634cde22fa508fedf7232abb1732c4122946f6aeeb5c8"
Oct 30 00:27:41.819477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.819441    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfz68\" (UniqueName: \"kubernetes.io/projected/cbd1c416-58d1-4259-aa8c-bbf63b887e3f-kube-api-access-jfz68\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:41.968129 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.921416    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95c0bd0-aeea-40db-923b-acddedcac7c6-kube-api-access-xn4fb\" (UniqueName: \"kubernetes.io/projected/a95c0bd0-aeea-40db-923b-acddedcac7c6-kube-api-access-xn4fb\") pod \"a95c0bd0-aeea-40db-923b-acddedcac7c6\" (UID: \"a95c0bd0-aeea-40db-923b-acddedcac7c6\") "
Oct 30 00:27:41.968129 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.921578    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a95c0bd0-aeea-40db-923b-acddedcac7c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95c0bd0-aeea-40db-923b-acddedcac7c6-plugins-config-volume\") pod \"a95c0bd0-aeea-40db-923b-acddedcac7c6\" (UID: \"a95c0bd0-aeea-40db-923b-acddedcac7c6\") "
Oct 30 00:27:41.972110 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.969583    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95c0bd0-aeea-40db-923b-acddedcac7c6-plugins-config-volume" pod "a95c0bd0-aeea-40db-923b-acddedcac7c6" (UID: "a95c0bd0-aeea-40db-923b-acddedcac7c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:27:41.972110 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.971233    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95c0bd0-aeea-40db-923b-acddedcac7c6-kube-api-access-xn4fb" pod "a95c0bd0-aeea-40db-923b-acddedcac7c6" (UID: "a95c0bd0-aeea-40db-923b-acddedcac7c6"). InnerVolumeSpecName "kube-api-access-xn4fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:27:41.974574 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.974550    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"]
Oct 30 00:27:41.981222 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:41.981198    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a15230-5cbc5c96db-sxmxv"]
Oct 30 00:27:42.048050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:42.029417    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn4fb\" (UniqueName: \"kubernetes.io/projected/a95c0bd0-aeea-40db-923b-acddedcac7c6-kube-api-access-xn4fb\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:42.048050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:42.029458    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95c0bd0-aeea-40db-923b-acddedcac7c6-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:27:42.509324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:42.509287    4441 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 00:27:42.509483 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:42.509342    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz" podUID="9a5b0544-84cb-494c-a40d-7b23774cc079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 00:27:42.611189 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:42.611148    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd1c416-58d1-4259-aa8c-bbf63b887e3f" path="/var/lib/kubelet/pods/cbd1c416-58d1-4259-aa8c-bbf63b887e3f/volumes"
Oct 30 00:27:42.771957 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:42.771876    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7" event={"ID":"a95c0bd0-aeea-40db-923b-acddedcac7c6","Type":"ContainerDied","Data":"ea034cdaec13038b64011d7525ddd44de5b19297548d1f6dfc8d1feb8fe2f55e"}
Oct 30 00:27:42.772390 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:42.772011    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"
Oct 30 00:27:42.856049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:42.849143    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"]
Oct 30 00:27:42.863567 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:42.863535    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-749bcfbdf9-2x6s7"]
Oct 30 00:27:43.516633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:43.516603    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:43.517324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:43.517294    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:43.518132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:43.518115    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:27:44.613422 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:44.613382    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a95c0bd0-aeea-40db-923b-acddedcac7c6" path="/var/lib/kubelet/pods/a95c0bd0-aeea-40db-923b-acddedcac7c6/volumes"
Oct 30 00:27:45.556933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:45.556904    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"
Oct 30 00:27:49.007260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.007225    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"]
Oct 30 00:27:49.008168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.008144    4441 cpu_manager.go:414] "containerMap: removing container" podUID="cbd1c416-58d1-4259-aa8c-bbf63b887e3f" containerName="payload-processing"
Oct 30 00:27:49.008168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.008164    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd1c416-58d1-4259-aa8c-bbf63b887e3f" containerName="payload-processing"
Oct 30 00:27:49.008333 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.008207    4441 cpu_manager.go:414] "containerMap: removing container" podUID="a95c0bd0-aeea-40db-923b-acddedcac7c6" containerName="payload-pre-processing"
Oct 30 00:27:49.008333 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.008213    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a95c0bd0-aeea-40db-923b-acddedcac7c6" containerName="payload-pre-processing"
Oct 30 00:27:49.008333 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.008315    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd1c416-58d1-4259-aa8c-bbf63b887e3f" containerName="payload-processing"
Oct 30 00:27:49.008333 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.008331    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a95c0bd0-aeea-40db-923b-acddedcac7c6" containerName="payload-pre-processing"
Oct 30 00:27:49.012401 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.012383    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.034266 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.034242    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-4500d5-serving-cert\""
Oct 30 00:27:49.039794 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.039768    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"]
Oct 30 00:27:49.080487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.080440    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv6fj\" (UniqueName: \"kubernetes.io/projected/aa997c6d-133a-4980-b6b9-aa5f2977af78-kube-api-access-jv6fj\") pod \"maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.080683 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.080605    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa997c6d-133a-4980-b6b9-aa5f2977af78-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.080766 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.080721    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-metrics-certs\") pod \"maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.080979 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.080911    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.182782 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.182748    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa997c6d-133a-4980-b6b9-aa5f2977af78-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.182984 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.182816    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-metrics-certs\") pod \"maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.182984 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.182950    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.183179 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.183103    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6fj\" (UniqueName: \"kubernetes.io/projected/aa997c6d-133a-4980-b6b9-aa5f2977af78-kube-api-access-jv6fj\") pod \"maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.183510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.183479    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa997c6d-133a-4980-b6b9-aa5f2977af78-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.185455 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.185432    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.185561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.185482    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-metrics-certs\") pod \"maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.205764 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.205728    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6fj\" (UniqueName: \"kubernetes.io/projected/aa997c6d-133a-4980-b6b9-aa5f2977af78-kube-api-access-jv6fj\") pod \"maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.328417 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.328330    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:49.650412 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:27:49.650369    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa997c6d_133a_4980_b6b9_aa5f2977af78.slice/crio-aed78ccc6f031ac7f705efd3de1e123c6f19ed3108ab33c6ea1d06c4d4d6d65b WatchSource:0}: Error finding container aed78ccc6f031ac7f705efd3de1e123c6f19ed3108ab33c6ea1d06c4d4d6d65b: Status 404 returned error can't find the container with id aed78ccc6f031ac7f705efd3de1e123c6f19ed3108ab33c6ea1d06c4d4d6d65b
Oct 30 00:27:49.652644 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.652616    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"]
Oct 30 00:27:49.839271 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:49.839231    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p" event={"ID":"aa997c6d-133a-4980-b6b9-aa5f2977af78","Type":"ContainerStarted","Data":"aed78ccc6f031ac7f705efd3de1e123c6f19ed3108ab33c6ea1d06c4d4d6d65b"}
Oct 30 00:27:50.853627 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:50.853006    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p" event={"ID":"aa997c6d-133a-4980-b6b9-aa5f2977af78","Type":"ContainerStarted","Data":"bb6e7c48f9012d19770084ac71411822a611814260648b5de600e045e317bca4"}
Oct 30 00:27:50.857154 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:50.857121    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:27:50.893801 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:50.893748    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p" podStartSLOduration=2.475021602 podStartE2EDuration="2.893732728s" totalImagesPullingTime="418.711126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:50.89095981 +0000 UTC m=+1380.416253687" watchObservedRunningTime="2026-10-30 00:27:50.893732728 +0000 UTC m=+1380.419026603"
Oct 30 00:27:57.876164 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:27:57.876130    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:28:02.546394 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:02.545492    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:28:03.902918 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:28:03.902884    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:28:05.091566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.091526    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"]
Oct 30 00:28:05.096613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.096590    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"
Oct 30 00:28:05.120815 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.120774    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"]
Oct 30 00:28:05.123694 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.123660    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a15230-dockercfg-7qhl7\""
Oct 30 00:28:05.124051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.124008    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a15230\""
Oct 30 00:28:05.228208 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.228164    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9v65\" (UniqueName: \"kubernetes.io/projected/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-kube-api-access-k9v65\") pod \"payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j\" (UID: \"1c7f7538-6f8f-4605-bac8-a0dfde5e4017\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"
Oct 30 00:28:05.228828 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.228791    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j\" (UID: \"1c7f7538-6f8f-4605-bac8-a0dfde5e4017\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"
Oct 30 00:28:05.330984 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.330955    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j\" (UID: \"1c7f7538-6f8f-4605-bac8-a0dfde5e4017\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"
Oct 30 00:28:05.331203 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.331160    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9v65\" (UniqueName: \"kubernetes.io/projected/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-kube-api-access-k9v65\") pod \"payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j\" (UID: \"1c7f7538-6f8f-4605-bac8-a0dfde5e4017\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"
Oct 30 00:28:05.331969 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.331942    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j\" (UID: \"1c7f7538-6f8f-4605-bac8-a0dfde5e4017\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"
Oct 30 00:28:05.355897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.355837    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9v65\" (UniqueName: \"kubernetes.io/projected/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-kube-api-access-k9v65\") pod \"payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j\" (UID: \"1c7f7538-6f8f-4605-bac8-a0dfde5e4017\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"
Oct 30 00:28:05.457416 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.457381    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"
Oct 30 00:28:05.819726 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:28:05.819696    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7f7538_6f8f_4605_bac8_a0dfde5e4017.slice/crio-153302d7e25110aaf6c422ecf4ba47585c68dfdb6a586a4f9a79bd7312273bf4 WatchSource:0}: Error finding container 153302d7e25110aaf6c422ecf4ba47585c68dfdb6a586a4f9a79bd7312273bf4: Status 404 returned error can't find the container with id 153302d7e25110aaf6c422ecf4ba47585c68dfdb6a586a4f9a79bd7312273bf4
Oct 30 00:28:05.820209 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.820185    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"]
Oct 30 00:28:05.993980 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.993943    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j" event={"ID":"1c7f7538-6f8f-4605-bac8-a0dfde5e4017","Type":"ContainerStarted","Data":"1d2343552e3befbf91cd86635296510feba80dbb984e25efb749dbb810210d21"}
Oct 30 00:28:05.994166 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:05.994000    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j" event={"ID":"1c7f7538-6f8f-4605-bac8-a0dfde5e4017","Type":"ContainerStarted","Data":"153302d7e25110aaf6c422ecf4ba47585c68dfdb6a586a4f9a79bd7312273bf4"}
Oct 30 00:28:06.032858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:06.032800    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j" podStartSLOduration=1.032785987 podStartE2EDuration="1.032785987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:28:06.029379928 +0000 UTC m=+1395.554673805" watchObservedRunningTime="2026-10-30 00:28:06.032785987 +0000 UTC m=+1395.558079864"
Oct 30 00:28:07.031139 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:07.031105    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"
Oct 30 00:28:18.041624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:18.041594    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"
Oct 30 00:28:25.547373 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:25.547342    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"]
Oct 30 00:28:30.548175 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:30.548103    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp" podUID="4bb99ba7-cf69-456b-8384-57ef12a418be" containerName="payload-processing" containerID="cri-o://524abb562cb4d2451040f24a2ef357dfe130925c4f54001a3338b92974de80ee" gracePeriod=25
Oct 30 00:28:30.860877 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:30.860845    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"
Oct 30 00:28:30.960326 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:30.960278    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb99ba7-cf69-456b-8384-57ef12a418be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb99ba7-cf69-456b-8384-57ef12a418be-plugins-config-volume\") pod \"4bb99ba7-cf69-456b-8384-57ef12a418be\" (UID: \"4bb99ba7-cf69-456b-8384-57ef12a418be\") "
Oct 30 00:28:30.960517 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:30.960443    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb99ba7-cf69-456b-8384-57ef12a418be-kube-api-access-ppfkk\" (UniqueName: \"kubernetes.io/projected/4bb99ba7-cf69-456b-8384-57ef12a418be-kube-api-access-ppfkk\") pod \"4bb99ba7-cf69-456b-8384-57ef12a418be\" (UID: \"4bb99ba7-cf69-456b-8384-57ef12a418be\") "
Oct 30 00:28:30.960746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:30.960720    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb99ba7-cf69-456b-8384-57ef12a418be-plugins-config-volume" pod "4bb99ba7-cf69-456b-8384-57ef12a418be" (UID: "4bb99ba7-cf69-456b-8384-57ef12a418be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:28:30.961275 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:30.961247    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb99ba7-cf69-456b-8384-57ef12a418be-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:28:30.962670 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:30.962644    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb99ba7-cf69-456b-8384-57ef12a418be-kube-api-access-ppfkk" pod "4bb99ba7-cf69-456b-8384-57ef12a418be" (UID: "4bb99ba7-cf69-456b-8384-57ef12a418be"). InnerVolumeSpecName "kube-api-access-ppfkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:28:31.062793 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:31.062752    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppfkk\" (UniqueName: \"kubernetes.io/projected/4bb99ba7-cf69-456b-8384-57ef12a418be-kube-api-access-ppfkk\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:28:31.240700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:31.240660    4441 generic.go:356] "Generic (PLEG): container finished" podID="4bb99ba7-cf69-456b-8384-57ef12a418be" containerID="524abb562cb4d2451040f24a2ef357dfe130925c4f54001a3338b92974de80ee" exitCode=0
Oct 30 00:28:31.240882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:31.240709    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"
Oct 30 00:28:31.240882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:31.240736    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp" event={"ID":"4bb99ba7-cf69-456b-8384-57ef12a418be","Type":"ContainerDied","Data":"524abb562cb4d2451040f24a2ef357dfe130925c4f54001a3338b92974de80ee"}
Oct 30 00:28:31.240882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:31.240794    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp" event={"ID":"4bb99ba7-cf69-456b-8384-57ef12a418be","Type":"ContainerDied","Data":"c6cf38b202b665ec49889cec6735a9e803bcaa234625c43ca5d59d076ba691c9"}
Oct 30 00:28:31.240882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:31.240824    4441 scope.go:122] "RemoveContainer" containerID="524abb562cb4d2451040f24a2ef357dfe130925c4f54001a3338b92974de80ee"
Oct 30 00:28:31.252551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:31.252532    4441 scope.go:122] "RemoveContainer" containerID="524abb562cb4d2451040f24a2ef357dfe130925c4f54001a3338b92974de80ee"
Oct 30 00:28:31.252872 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:28:31.252849    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524abb562cb4d2451040f24a2ef357dfe130925c4f54001a3338b92974de80ee\": container with ID starting with 524abb562cb4d2451040f24a2ef357dfe130925c4f54001a3338b92974de80ee not found: ID does not exist" containerID="524abb562cb4d2451040f24a2ef357dfe130925c4f54001a3338b92974de80ee"
Oct 30 00:28:31.252941 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:31.252883    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524abb562cb4d2451040f24a2ef357dfe130925c4f54001a3338b92974de80ee"} err="failed to get container status \"524abb562cb4d2451040f24a2ef357dfe130925c4f54001a3338b92974de80ee\": rpc error: code = NotFound desc = could not find container \"524abb562cb4d2451040f24a2ef357dfe130925c4f54001a3338b92974de80ee\": container with ID starting with 524abb562cb4d2451040f24a2ef357dfe130925c4f54001a3338b92974de80ee not found: ID does not exist"
Oct 30 00:28:31.288173 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:31.288145    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"]
Oct 30 00:28:31.292121 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:31.292095    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cfcf7-86b9b66548-494bp"]
Oct 30 00:28:32.611631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:32.611601    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb99ba7-cf69-456b-8384-57ef12a418be" path="/var/lib/kubelet/pods/4bb99ba7-cf69-456b-8384-57ef12a418be/volumes"
Oct 30 00:28:33.912752 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:28:33.912701    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:28:35.605893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:35.605855    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"]
Oct 30 00:28:35.660696 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:35.660664    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"]
Oct 30 00:28:40.606382 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:40.606312    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc" podUID="089e7957-fa4a-4bf4-b457-28af567838fd" containerName="payload-processing" containerID="cri-o://8069a8669c576dc257314da70fada451665f14046e8f916625ce4f1eeb15296e" gracePeriod=25
Oct 30 00:28:40.661955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:40.661873    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm" podUID="663f591e-ad15-44c2-9983-5464126875c2" containerName="payload-pre-processing" containerID="cri-o://ae23571b87b0bc2fdc6173febbb926b852e4a1d2a05da9f85d913e5538ac1a37" gracePeriod=25
Oct 30 00:28:41.035718 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.035692    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"
Oct 30 00:28:41.078041 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.078005    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"
Oct 30 00:28:41.094195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.093728    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663f591e-ad15-44c2-9983-5464126875c2-kube-api-access-b6v6p\" (UniqueName: \"kubernetes.io/projected/663f591e-ad15-44c2-9983-5464126875c2-kube-api-access-b6v6p\") pod \"663f591e-ad15-44c2-9983-5464126875c2\" (UID: \"663f591e-ad15-44c2-9983-5464126875c2\") "
Oct 30 00:28:41.094195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.093892    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/089e7957-fa4a-4bf4-b457-28af567838fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089e7957-fa4a-4bf4-b457-28af567838fd-plugins-config-volume\") pod \"089e7957-fa4a-4bf4-b457-28af567838fd\" (UID: \"089e7957-fa4a-4bf4-b457-28af567838fd\") "
Oct 30 00:28:41.094195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.093921    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663f591e-ad15-44c2-9983-5464126875c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663f591e-ad15-44c2-9983-5464126875c2-plugins-config-volume\") pod \"663f591e-ad15-44c2-9983-5464126875c2\" (UID: \"663f591e-ad15-44c2-9983-5464126875c2\") "
Oct 30 00:28:41.094518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.094276    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089e7957-fa4a-4bf4-b457-28af567838fd-plugins-config-volume" pod "089e7957-fa4a-4bf4-b457-28af567838fd" (UID: "089e7957-fa4a-4bf4-b457-28af567838fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:28:41.094781 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.094754    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663f591e-ad15-44c2-9983-5464126875c2-plugins-config-volume" pod "663f591e-ad15-44c2-9983-5464126875c2" (UID: "663f591e-ad15-44c2-9983-5464126875c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:28:41.094935 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.094916    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089e7957-fa4a-4bf4-b457-28af567838fd-kube-api-access-4q6hb\" (UniqueName: \"kubernetes.io/projected/089e7957-fa4a-4bf4-b457-28af567838fd-kube-api-access-4q6hb\") pod \"089e7957-fa4a-4bf4-b457-28af567838fd\" (UID: \"089e7957-fa4a-4bf4-b457-28af567838fd\") "
Oct 30 00:28:41.095649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.095625    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089e7957-fa4a-4bf4-b457-28af567838fd-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:28:41.095649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.095649    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663f591e-ad15-44c2-9983-5464126875c2-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:28:41.097616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.097594    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089e7957-fa4a-4bf4-b457-28af567838fd-kube-api-access-4q6hb" pod "089e7957-fa4a-4bf4-b457-28af567838fd" (UID: "089e7957-fa4a-4bf4-b457-28af567838fd"). InnerVolumeSpecName "kube-api-access-4q6hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:28:41.097716 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.097652    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663f591e-ad15-44c2-9983-5464126875c2-kube-api-access-b6v6p" pod "663f591e-ad15-44c2-9983-5464126875c2" (UID: "663f591e-ad15-44c2-9983-5464126875c2"). InnerVolumeSpecName "kube-api-access-b6v6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:28:41.197178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.197094    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q6hb\" (UniqueName: \"kubernetes.io/projected/089e7957-fa4a-4bf4-b457-28af567838fd-kube-api-access-4q6hb\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:28:41.197178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.197131    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6v6p\" (UniqueName: \"kubernetes.io/projected/663f591e-ad15-44c2-9983-5464126875c2-kube-api-access-b6v6p\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:28:41.372436 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.372401    4441 generic.go:356] "Generic (PLEG): container finished" podID="663f591e-ad15-44c2-9983-5464126875c2" containerID="ae23571b87b0bc2fdc6173febbb926b852e4a1d2a05da9f85d913e5538ac1a37" exitCode=0
Oct 30 00:28:41.372620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.372455    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"
Oct 30 00:28:41.372620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.372480    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm" event={"ID":"663f591e-ad15-44c2-9983-5464126875c2","Type":"ContainerDied","Data":"ae23571b87b0bc2fdc6173febbb926b852e4a1d2a05da9f85d913e5538ac1a37"}
Oct 30 00:28:41.372620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.372536    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm" event={"ID":"663f591e-ad15-44c2-9983-5464126875c2","Type":"ContainerDied","Data":"7b809d52c6b8b2bb331dbca1c3cde680f0406244c584e3abea10bbeab63c22d4"}
Oct 30 00:28:41.372620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.372567    4441 scope.go:122] "RemoveContainer" containerID="ae23571b87b0bc2fdc6173febbb926b852e4a1d2a05da9f85d913e5538ac1a37"
Oct 30 00:28:41.374246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.374220    4441 generic.go:356] "Generic (PLEG): container finished" podID="089e7957-fa4a-4bf4-b457-28af567838fd" containerID="8069a8669c576dc257314da70fada451665f14046e8f916625ce4f1eeb15296e" exitCode=0
Oct 30 00:28:41.374369 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.374273    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc" event={"ID":"089e7957-fa4a-4bf4-b457-28af567838fd","Type":"ContainerDied","Data":"8069a8669c576dc257314da70fada451665f14046e8f916625ce4f1eeb15296e"}
Oct 30 00:28:41.374369 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.374319    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc" event={"ID":"089e7957-fa4a-4bf4-b457-28af567838fd","Type":"ContainerDied","Data":"c6ea9a9c37bf4bd4dadc4fc6b81b9c2a381d9f4bb3cb7fcc9e2e2a8f93f22fce"}
Oct 30 00:28:41.374369 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.374277    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"
Oct 30 00:28:41.386602 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.386583    4441 scope.go:122] "RemoveContainer" containerID="ae23571b87b0bc2fdc6173febbb926b852e4a1d2a05da9f85d913e5538ac1a37"
Oct 30 00:28:41.386938 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:28:41.386893    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae23571b87b0bc2fdc6173febbb926b852e4a1d2a05da9f85d913e5538ac1a37\": container with ID starting with ae23571b87b0bc2fdc6173febbb926b852e4a1d2a05da9f85d913e5538ac1a37 not found: ID does not exist" containerID="ae23571b87b0bc2fdc6173febbb926b852e4a1d2a05da9f85d913e5538ac1a37"
Oct 30 00:28:41.387087 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.386946    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae23571b87b0bc2fdc6173febbb926b852e4a1d2a05da9f85d913e5538ac1a37"} err="failed to get container status \"ae23571b87b0bc2fdc6173febbb926b852e4a1d2a05da9f85d913e5538ac1a37\": rpc error: code = NotFound desc = could not find container \"ae23571b87b0bc2fdc6173febbb926b852e4a1d2a05da9f85d913e5538ac1a37\": container with ID starting with ae23571b87b0bc2fdc6173febbb926b852e4a1d2a05da9f85d913e5538ac1a37 not found: ID does not exist"
Oct 30 00:28:41.387087 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.386972    4441 scope.go:122] "RemoveContainer" containerID="8069a8669c576dc257314da70fada451665f14046e8f916625ce4f1eeb15296e"
Oct 30 00:28:41.401203 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.401183    4441 scope.go:122] "RemoveContainer" containerID="8069a8669c576dc257314da70fada451665f14046e8f916625ce4f1eeb15296e"
Oct 30 00:28:41.404254 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:28:41.404226    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8069a8669c576dc257314da70fada451665f14046e8f916625ce4f1eeb15296e\": container with ID starting with 8069a8669c576dc257314da70fada451665f14046e8f916625ce4f1eeb15296e not found: ID does not exist" containerID="8069a8669c576dc257314da70fada451665f14046e8f916625ce4f1eeb15296e"
Oct 30 00:28:41.404340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.404261    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8069a8669c576dc257314da70fada451665f14046e8f916625ce4f1eeb15296e"} err="failed to get container status \"8069a8669c576dc257314da70fada451665f14046e8f916625ce4f1eeb15296e\": rpc error: code = NotFound desc = could not find container \"8069a8669c576dc257314da70fada451665f14046e8f916625ce4f1eeb15296e\": container with ID starting with 8069a8669c576dc257314da70fada451665f14046e8f916625ce4f1eeb15296e not found: ID does not exist"
Oct 30 00:28:41.431615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.431589    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"]
Oct 30 00:28:41.435240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.435217    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-bb16a8-6d5c6c4548-h2jhc"]
Oct 30 00:28:41.574084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.568153    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"]
Oct 30 00:28:41.590708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:41.589368    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-bb16a8-6f96487f75-727lm"]
Oct 30 00:28:42.612473 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:42.612444    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089e7957-fa4a-4bf4-b457-28af567838fd" path="/var/lib/kubelet/pods/089e7957-fa4a-4bf4-b457-28af567838fd/volumes"
Oct 30 00:28:42.613093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:42.613073    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663f591e-ad15-44c2-9983-5464126875c2" path="/var/lib/kubelet/pods/663f591e-ad15-44c2-9983-5464126875c2/volumes"
Oct 30 00:28:43.029550 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:43.029514    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:28:51.819009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:51.818974    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"]
Oct 30 00:28:51.819580 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:51.819179    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2" podUID="c554ae97-8f0b-4bf9-bf77-6ac3bfb42575" containerName="payload-processing" containerID="cri-o://dbbf68baf0a49135a7b67b62cfcdfceeba4f33a8d0ed63631a36a3b0a8511387" gracePeriod=30
Oct 30 00:28:52.128093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.128070    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"
Oct 30 00:28:52.258552 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.257555    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-kube-api-access-hcvrw\" (UniqueName: \"kubernetes.io/projected/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-kube-api-access-hcvrw\") pod \"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575\" (UID: \"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575\") "
Oct 30 00:28:52.258552 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.257777    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-plugins-config-volume\") pod \"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575\" (UID: \"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575\") "
Oct 30 00:28:52.275322 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.275282    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-plugins-config-volume" pod "c554ae97-8f0b-4bf9-bf77-6ac3bfb42575" (UID: "c554ae97-8f0b-4bf9-bf77-6ac3bfb42575"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:28:52.296052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.293215    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-kube-api-access-hcvrw" pod "c554ae97-8f0b-4bf9-bf77-6ac3bfb42575" (UID: "c554ae97-8f0b-4bf9-bf77-6ac3bfb42575"). InnerVolumeSpecName "kube-api-access-hcvrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:28:52.354925 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.352664    4441 scope.go:122] "RemoveContainer" containerID="0a2f21822a04a80ef3e54e47bf9bdab5f353b4317c81226ca89e8e203e5eb6b1"
Oct 30 00:28:52.376268 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.376241    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcvrw\" (UniqueName: \"kubernetes.io/projected/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-kube-api-access-hcvrw\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:28:52.376268 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.376266    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:28:52.445254 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.445222    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"]
Oct 30 00:28:52.454451 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.454418    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"]
Oct 30 00:28:52.492592 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.492554    4441 generic.go:356] "Generic (PLEG): container finished" podID="c554ae97-8f0b-4bf9-bf77-6ac3bfb42575" containerID="dbbf68baf0a49135a7b67b62cfcdfceeba4f33a8d0ed63631a36a3b0a8511387" exitCode=0
Oct 30 00:28:52.492773 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.492598    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2" event={"ID":"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575","Type":"ContainerDied","Data":"dbbf68baf0a49135a7b67b62cfcdfceeba4f33a8d0ed63631a36a3b0a8511387"}
Oct 30 00:28:52.492773 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.492675    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"
Oct 30 00:28:52.492773 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.492684    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2" event={"ID":"c554ae97-8f0b-4bf9-bf77-6ac3bfb42575","Type":"ContainerDied","Data":"48b95683db44c8f6d5f071979890a597f68d791bc6267248f178eb7e75ccace0"}
Oct 30 00:28:52.492773 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.492736    4441 scope.go:122] "RemoveContainer" containerID="dbbf68baf0a49135a7b67b62cfcdfceeba4f33a8d0ed63631a36a3b0a8511387"
Oct 30 00:28:52.535047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.529151    4441 scope.go:122] "RemoveContainer" containerID="e6f7b78f32afa089089f3b5d7f9ab99c9e81cf084bcf926d7df5881503abf659"
Oct 30 00:28:52.547703 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.547678    4441 scope.go:122] "RemoveContainer" containerID="dbbf68baf0a49135a7b67b62cfcdfceeba4f33a8d0ed63631a36a3b0a8511387"
Oct 30 00:28:52.548423 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:28:52.548399    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbf68baf0a49135a7b67b62cfcdfceeba4f33a8d0ed63631a36a3b0a8511387\": container with ID starting with dbbf68baf0a49135a7b67b62cfcdfceeba4f33a8d0ed63631a36a3b0a8511387 not found: ID does not exist" containerID="dbbf68baf0a49135a7b67b62cfcdfceeba4f33a8d0ed63631a36a3b0a8511387"
Oct 30 00:28:52.548745 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.548695    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbf68baf0a49135a7b67b62cfcdfceeba4f33a8d0ed63631a36a3b0a8511387"} err="failed to get container status \"dbbf68baf0a49135a7b67b62cfcdfceeba4f33a8d0ed63631a36a3b0a8511387\": rpc error: code = NotFound desc = could not find container \"dbbf68baf0a49135a7b67b62cfcdfceeba4f33a8d0ed63631a36a3b0a8511387\": container with ID starting with dbbf68baf0a49135a7b67b62cfcdfceeba4f33a8d0ed63631a36a3b0a8511387 not found: ID does not exist"
Oct 30 00:28:52.549060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.549045    4441 scope.go:122] "RemoveContainer" containerID="e6f7b78f32afa089089f3b5d7f9ab99c9e81cf084bcf926d7df5881503abf659"
Oct 30 00:28:52.549945 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:28:52.549921    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f7b78f32afa089089f3b5d7f9ab99c9e81cf084bcf926d7df5881503abf659\": container with ID starting with e6f7b78f32afa089089f3b5d7f9ab99c9e81cf084bcf926d7df5881503abf659 not found: ID does not exist" containerID="e6f7b78f32afa089089f3b5d7f9ab99c9e81cf084bcf926d7df5881503abf659"
Oct 30 00:28:52.550106 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.550083    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f7b78f32afa089089f3b5d7f9ab99c9e81cf084bcf926d7df5881503abf659"} err="failed to get container status \"e6f7b78f32afa089089f3b5d7f9ab99c9e81cf084bcf926d7df5881503abf659\": rpc error: code = NotFound desc = could not find container \"e6f7b78f32afa089089f3b5d7f9ab99c9e81cf084bcf926d7df5881503abf659\": container with ID starting with e6f7b78f32afa089089f3b5d7f9ab99c9e81cf084bcf926d7df5881503abf659 not found: ID does not exist"
Oct 30 00:28:52.593052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.587625    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"]
Oct 30 00:28:52.593052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.587683    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f8397b-5c957f784d-lgpp2"]
Oct 30 00:28:52.631840 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:52.631762    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c554ae97-8f0b-4bf9-bf77-6ac3bfb42575" path="/var/lib/kubelet/pods/c554ae97-8f0b-4bf9-bf77-6ac3bfb42575/volumes"
Oct 30 00:28:57.445710 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.445665    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb" podUID="71a1c040-83ea-4f04-b8a3-e3f3c59937d9" containerName="payload-processing" containerID="cri-o://fe745c27e24224794999671dd85362f3d51814c8cd024e6f7e335c71e5b99d18" gracePeriod=25
Oct 30 00:28:57.455289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.455253    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n" podUID="1f51de93-61c4-40c5-808f-347d21189d3d" containerName="payload-pre-processing" containerID="cri-o://095bdfefc4ca838c71449bd37ca8f95fce0fed63a95986534e75655865e1530a" gracePeriod=25
Oct 30 00:28:57.790134 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.790109    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"
Oct 30 00:28:57.793692 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.793671    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"
Oct 30 00:28:57.872655 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.872618    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f51de93-61c4-40c5-808f-347d21189d3d-kube-api-access-pxvpp\" (UniqueName: \"kubernetes.io/projected/1f51de93-61c4-40c5-808f-347d21189d3d-kube-api-access-pxvpp\") pod \"1f51de93-61c4-40c5-808f-347d21189d3d\" (UID: \"1f51de93-61c4-40c5-808f-347d21189d3d\") "
Oct 30 00:28:57.873293 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.873272    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f51de93-61c4-40c5-808f-347d21189d3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f51de93-61c4-40c5-808f-347d21189d3d-plugins-config-volume\") pod \"1f51de93-61c4-40c5-808f-347d21189d3d\" (UID: \"1f51de93-61c4-40c5-808f-347d21189d3d\") "
Oct 30 00:28:57.873794 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.873770    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f51de93-61c4-40c5-808f-347d21189d3d-plugins-config-volume" pod "1f51de93-61c4-40c5-808f-347d21189d3d" (UID: "1f51de93-61c4-40c5-808f-347d21189d3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:28:57.873945 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.873920    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-plugins-config-volume" pod "71a1c040-83ea-4f04-b8a3-e3f3c59937d9" (UID: "71a1c040-83ea-4f04-b8a3-e3f3c59937d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:28:57.874013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.873963    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-plugins-config-volume\") pod \"71a1c040-83ea-4f04-b8a3-e3f3c59937d9\" (UID: \"71a1c040-83ea-4f04-b8a3-e3f3c59937d9\") "
Oct 30 00:28:57.874088 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.874074    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-kube-api-access-xsxx4\" (UniqueName: \"kubernetes.io/projected/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-kube-api-access-xsxx4\") pod \"71a1c040-83ea-4f04-b8a3-e3f3c59937d9\" (UID: \"71a1c040-83ea-4f04-b8a3-e3f3c59937d9\") "
Oct 30 00:28:57.876217 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.876191    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-kube-api-access-xsxx4" pod "71a1c040-83ea-4f04-b8a3-e3f3c59937d9" (UID: "71a1c040-83ea-4f04-b8a3-e3f3c59937d9"). InnerVolumeSpecName "kube-api-access-xsxx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:28:57.876494 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.876471    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f51de93-61c4-40c5-808f-347d21189d3d-kube-api-access-pxvpp" pod "1f51de93-61c4-40c5-808f-347d21189d3d" (UID: "1f51de93-61c4-40c5-808f-347d21189d3d"). InnerVolumeSpecName "kube-api-access-pxvpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:28:57.876593 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.876535    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f51de93-61c4-40c5-808f-347d21189d3d-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:28:57.876593 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.876557    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:28:57.876593 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.876571    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsxx4\" (UniqueName: \"kubernetes.io/projected/71a1c040-83ea-4f04-b8a3-e3f3c59937d9-kube-api-access-xsxx4\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:28:57.978215 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:57.978183    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxvpp\" (UniqueName: \"kubernetes.io/projected/1f51de93-61c4-40c5-808f-347d21189d3d-kube-api-access-pxvpp\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:28:58.546017 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.545983    4441 generic.go:356] "Generic (PLEG): container finished" podID="1f51de93-61c4-40c5-808f-347d21189d3d" containerID="095bdfefc4ca838c71449bd37ca8f95fce0fed63a95986534e75655865e1530a" exitCode=0
Oct 30 00:28:58.546477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.546046    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"
Oct 30 00:28:58.546477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.546076    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n" event={"ID":"1f51de93-61c4-40c5-808f-347d21189d3d","Type":"ContainerDied","Data":"095bdfefc4ca838c71449bd37ca8f95fce0fed63a95986534e75655865e1530a"}
Oct 30 00:28:58.546477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.546158    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n" event={"ID":"1f51de93-61c4-40c5-808f-347d21189d3d","Type":"ContainerDied","Data":"49b7217f8e03b4cd3a7dc5ee5c10146e0be03e1a75b2caadfe899ea60a2904be"}
Oct 30 00:28:58.546477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.546180    4441 scope.go:122] "RemoveContainer" containerID="095bdfefc4ca838c71449bd37ca8f95fce0fed63a95986534e75655865e1530a"
Oct 30 00:28:58.547774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.547753    4441 generic.go:356] "Generic (PLEG): container finished" podID="71a1c040-83ea-4f04-b8a3-e3f3c59937d9" containerID="fe745c27e24224794999671dd85362f3d51814c8cd024e6f7e335c71e5b99d18" exitCode=0
Oct 30 00:28:58.547864 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.547793    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"
Oct 30 00:28:58.547928 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.547871    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb" event={"ID":"71a1c040-83ea-4f04-b8a3-e3f3c59937d9","Type":"ContainerDied","Data":"fe745c27e24224794999671dd85362f3d51814c8cd024e6f7e335c71e5b99d18"}
Oct 30 00:28:58.547928 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.547907    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb" event={"ID":"71a1c040-83ea-4f04-b8a3-e3f3c59937d9","Type":"ContainerDied","Data":"7f41f38d02fa35344b5d68339a72af49f85b814edeaa13b1aaca5e1832cb5048"}
Oct 30 00:28:58.560070 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.560010    4441 scope.go:122] "RemoveContainer" containerID="095bdfefc4ca838c71449bd37ca8f95fce0fed63a95986534e75655865e1530a"
Oct 30 00:28:58.560391 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:28:58.560360    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"095bdfefc4ca838c71449bd37ca8f95fce0fed63a95986534e75655865e1530a\": container with ID starting with 095bdfefc4ca838c71449bd37ca8f95fce0fed63a95986534e75655865e1530a not found: ID does not exist" containerID="095bdfefc4ca838c71449bd37ca8f95fce0fed63a95986534e75655865e1530a"
Oct 30 00:28:58.560457 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.560401    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095bdfefc4ca838c71449bd37ca8f95fce0fed63a95986534e75655865e1530a"} err="failed to get container status \"095bdfefc4ca838c71449bd37ca8f95fce0fed63a95986534e75655865e1530a\": rpc error: code = NotFound desc = could not find container \"095bdfefc4ca838c71449bd37ca8f95fce0fed63a95986534e75655865e1530a\": container with ID starting with 095bdfefc4ca838c71449bd37ca8f95fce0fed63a95986534e75655865e1530a not found: ID does not exist"
Oct 30 00:28:58.560457 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.560423    4441 scope.go:122] "RemoveContainer" containerID="fe745c27e24224794999671dd85362f3d51814c8cd024e6f7e335c71e5b99d18"
Oct 30 00:28:58.571072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.571054    4441 scope.go:122] "RemoveContainer" containerID="fe745c27e24224794999671dd85362f3d51814c8cd024e6f7e335c71e5b99d18"
Oct 30 00:28:58.571356 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:28:58.571333    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe745c27e24224794999671dd85362f3d51814c8cd024e6f7e335c71e5b99d18\": container with ID starting with fe745c27e24224794999671dd85362f3d51814c8cd024e6f7e335c71e5b99d18 not found: ID does not exist" containerID="fe745c27e24224794999671dd85362f3d51814c8cd024e6f7e335c71e5b99d18"
Oct 30 00:28:58.571425 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.571364    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe745c27e24224794999671dd85362f3d51814c8cd024e6f7e335c71e5b99d18"} err="failed to get container status \"fe745c27e24224794999671dd85362f3d51814c8cd024e6f7e335c71e5b99d18\": rpc error: code = NotFound desc = could not find container \"fe745c27e24224794999671dd85362f3d51814c8cd024e6f7e335c71e5b99d18\": container with ID starting with fe745c27e24224794999671dd85362f3d51814c8cd024e6f7e335c71e5b99d18 not found: ID does not exist"
Oct 30 00:28:58.610821 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.610795    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"]
Oct 30 00:28:58.613174 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.613151    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-00df5f-6b86f99b5f-8b46n"]
Oct 30 00:28:58.637197 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.637173    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"]
Oct 30 00:28:58.640996 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:28:58.640975    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-00df5f-7b75d75d4d-cv6zb"]
Oct 30 00:29:00.610309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:00.610275    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f51de93-61c4-40c5-808f-347d21189d3d" path="/var/lib/kubelet/pods/1f51de93-61c4-40c5-808f-347d21189d3d/volumes"
Oct 30 00:29:00.610711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:00.610696    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a1c040-83ea-4f04-b8a3-e3f3c59937d9" path="/var/lib/kubelet/pods/71a1c040-83ea-4f04-b8a3-e3f3c59937d9/volumes"
Oct 30 00:29:03.902576 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:29:03.902535    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:29:11.699776 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:11.699630    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"]
Oct 30 00:29:11.700408 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:11.700358    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz" podUID="9a5b0544-84cb-494c-a40d-7b23774cc079" containerName="istio-proxy" containerID="cri-o://fc686a206abcba3c8a4ed2b27ab1f5d51d8049ac1e79c16d3c94142461ba7fa7" gracePeriod=30
Oct 30 00:29:13.520894 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:13.520855    4441 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:29:13.521322 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:13.520920    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz" podUID="9a5b0544-84cb-494c-a40d-7b23774cc079" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:29:14.406961 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:14.406924    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"]
Oct 30 00:29:14.407233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:14.407207    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy" containerID="cri-o://a23b8be31f268771894461bd59f3a641792f43a2489b694846ef35d6a380c23d" gracePeriod=30
Oct 30 00:29:17.106476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.104289    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:29:17.177320 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.177288    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-token\" (UniqueName: \"kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-token\") pod \"9a5b0544-84cb-494c-a40d-7b23774cc079\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") "
Oct 30 00:29:17.178205 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.178177    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-certs\") pod \"9a5b0544-84cb-494c-a40d-7b23774cc079\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") "
Oct 30 00:29:17.178526 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.178506    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-podinfo\") pod \"9a5b0544-84cb-494c-a40d-7b23774cc079\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") "
Oct 30 00:29:17.178659 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.178645    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-socket\") pod \"9a5b0544-84cb-494c-a40d-7b23774cc079\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") "
Oct 30 00:29:17.178787 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.178772    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-kube-api-access-gtlxs\" (UniqueName: \"kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-kube-api-access-gtlxs\") pod \"9a5b0544-84cb-494c-a40d-7b23774cc079\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") "
Oct 30 00:29:17.178904 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.178889    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-data\") pod \"9a5b0544-84cb-494c-a40d-7b23774cc079\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") "
Oct 30 00:29:17.179089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.179017    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a5b0544-84cb-494c-a40d-7b23774cc079-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5b0544-84cb-494c-a40d-7b23774cc079-istiod-ca-cert\") pod \"9a5b0544-84cb-494c-a40d-7b23774cc079\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") "
Oct 30 00:29:17.179276 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.179260    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-credential-socket\") pod \"9a5b0544-84cb-494c-a40d-7b23774cc079\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") "
Oct 30 00:29:17.179501 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.179485    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-envoy\") pod \"9a5b0544-84cb-494c-a40d-7b23774cc079\" (UID: \"9a5b0544-84cb-494c-a40d-7b23774cc079\") "
Oct 30 00:29:17.180343 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.179498    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-socket" pod "9a5b0544-84cb-494c-a40d-7b23774cc079" (UID: "9a5b0544-84cb-494c-a40d-7b23774cc079"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:29:17.180343 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.179709    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-certs" pod "9a5b0544-84cb-494c-a40d-7b23774cc079" (UID: "9a5b0544-84cb-494c-a40d-7b23774cc079"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:29:17.180503 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.180133    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-credential-socket" pod "9a5b0544-84cb-494c-a40d-7b23774cc079" (UID: "9a5b0544-84cb-494c-a40d-7b23774cc079"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:29:17.180503 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.180247    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-token" pod "9a5b0544-84cb-494c-a40d-7b23774cc079" (UID: "9a5b0544-84cb-494c-a40d-7b23774cc079"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:29:17.180503 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.180291    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-data" pod "9a5b0544-84cb-494c-a40d-7b23774cc079" (UID: "9a5b0544-84cb-494c-a40d-7b23774cc079"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:29:17.180503 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.180319    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5b0544-84cb-494c-a40d-7b23774cc079-istiod-ca-cert" pod "9a5b0544-84cb-494c-a40d-7b23774cc079" (UID: "9a5b0544-84cb-494c-a40d-7b23774cc079"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:29:17.180865 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.180845    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:17.181294 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.181275    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:17.181411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.181398    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:17.181502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.181490    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:17.182387 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.182356    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-envoy" pod "9a5b0544-84cb-494c-a40d-7b23774cc079" (UID: "9a5b0544-84cb-494c-a40d-7b23774cc079"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:29:17.183607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.183587    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-kube-api-access-gtlxs" pod "9a5b0544-84cb-494c-a40d-7b23774cc079" (UID: "9a5b0544-84cb-494c-a40d-7b23774cc079"). InnerVolumeSpecName "kube-api-access-gtlxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:29:17.183746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.183728    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-podinfo" pod "9a5b0544-84cb-494c-a40d-7b23774cc079" (UID: "9a5b0544-84cb-494c-a40d-7b23774cc079"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:29:17.283639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.283599    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:17.283639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.283641    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtlxs\" (UniqueName: \"kubernetes.io/projected/9a5b0544-84cb-494c-a40d-7b23774cc079-kube-api-access-gtlxs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:17.283872 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.283657    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:17.283872 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.283681    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5b0544-84cb-494c-a40d-7b23774cc079-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:17.283872 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.283703    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5b0544-84cb-494c-a40d-7b23774cc079-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:17.734466 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.734424    4441 generic.go:356] "Generic (PLEG): container finished" podID="9a5b0544-84cb-494c-a40d-7b23774cc079" containerID="fc686a206abcba3c8a4ed2b27ab1f5d51d8049ac1e79c16d3c94142461ba7fa7" exitCode=0
Oct 30 00:29:17.734668 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.734483    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"
Oct 30 00:29:17.734737 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.734702    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz" event={"ID":"9a5b0544-84cb-494c-a40d-7b23774cc079","Type":"ContainerDied","Data":"fc686a206abcba3c8a4ed2b27ab1f5d51d8049ac1e79c16d3c94142461ba7fa7"}
Oct 30 00:29:17.734795 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.734755    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz" event={"ID":"9a5b0544-84cb-494c-a40d-7b23774cc079","Type":"ContainerDied","Data":"0eb03b94d306c067ae6e85685e268b02551f102a61aea489797522bdf7a17404"}
Oct 30 00:29:17.734795 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.734789    4441 scope.go:122] "RemoveContainer" containerID="fc686a206abcba3c8a4ed2b27ab1f5d51d8049ac1e79c16d3c94142461ba7fa7"
Oct 30 00:29:17.748626 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.748610    4441 scope.go:122] "RemoveContainer" containerID="fc686a206abcba3c8a4ed2b27ab1f5d51d8049ac1e79c16d3c94142461ba7fa7"
Oct 30 00:29:17.748943 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:29:17.748922    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc686a206abcba3c8a4ed2b27ab1f5d51d8049ac1e79c16d3c94142461ba7fa7\": container with ID starting with fc686a206abcba3c8a4ed2b27ab1f5d51d8049ac1e79c16d3c94142461ba7fa7 not found: ID does not exist" containerID="fc686a206abcba3c8a4ed2b27ab1f5d51d8049ac1e79c16d3c94142461ba7fa7"
Oct 30 00:29:17.749042 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.748953    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc686a206abcba3c8a4ed2b27ab1f5d51d8049ac1e79c16d3c94142461ba7fa7"} err="failed to get container status \"fc686a206abcba3c8a4ed2b27ab1f5d51d8049ac1e79c16d3c94142461ba7fa7\": rpc error: code = NotFound desc = could not find container \"fc686a206abcba3c8a4ed2b27ab1f5d51d8049ac1e79c16d3c94142461ba7fa7\": container with ID starting with fc686a206abcba3c8a4ed2b27ab1f5d51d8049ac1e79c16d3c94142461ba7fa7 not found: ID does not exist"
Oct 30 00:29:17.790969 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.790945    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"]
Oct 30 00:29:17.800487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:17.800461    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-9zmgz"]
Oct 30 00:29:18.610218 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:18.610190    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a5b0544-84cb-494c-a40d-7b23774cc079" path="/var/lib/kubelet/pods/9a5b0544-84cb-494c-a40d-7b23774cc079/volumes"
Oct 30 00:29:19.710872 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.710849    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:29:19.776933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.776898    4441 generic.go:356] "Generic (PLEG): container finished" podID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerID="a23b8be31f268771894461bd59f3a641792f43a2489b694846ef35d6a380c23d" exitCode=0
Oct 30 00:29:19.777162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.776965    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"
Oct 30 00:29:19.777162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.777083    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" event={"ID":"4a972f67-aa0b-4444-9b0a-9662f67e6cac","Type":"ContainerDied","Data":"a23b8be31f268771894461bd59f3a641792f43a2489b694846ef35d6a380c23d"}
Oct 30 00:29:19.777162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.777155    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2" event={"ID":"4a972f67-aa0b-4444-9b0a-9662f67e6cac","Type":"ContainerDied","Data":"3ba47d656806816d102dd2e2383c16921def39984bc92b70b933f086f3f22770"}
Oct 30 00:29:19.777351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.777185    4441 scope.go:122] "RemoveContainer" containerID="a23b8be31f268771894461bd59f3a641792f43a2489b694846ef35d6a380c23d"
Oct 30 00:29:19.789457 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.789428    4441 scope.go:122] "RemoveContainer" containerID="58987af5c64e3fa80dd4753614ce45bfded62887cdb075f2e662b7b95bf94211"
Oct 30 00:29:19.799427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.799392    4441 scope.go:122] "RemoveContainer" containerID="a23b8be31f268771894461bd59f3a641792f43a2489b694846ef35d6a380c23d"
Oct 30 00:29:19.799735 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:29:19.799714    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23b8be31f268771894461bd59f3a641792f43a2489b694846ef35d6a380c23d\": container with ID starting with a23b8be31f268771894461bd59f3a641792f43a2489b694846ef35d6a380c23d not found: ID does not exist" containerID="a23b8be31f268771894461bd59f3a641792f43a2489b694846ef35d6a380c23d"
Oct 30 00:29:19.799829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.799741    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23b8be31f268771894461bd59f3a641792f43a2489b694846ef35d6a380c23d"} err="failed to get container status \"a23b8be31f268771894461bd59f3a641792f43a2489b694846ef35d6a380c23d\": rpc error: code = NotFound desc = could not find container \"a23b8be31f268771894461bd59f3a641792f43a2489b694846ef35d6a380c23d\": container with ID starting with a23b8be31f268771894461bd59f3a641792f43a2489b694846ef35d6a380c23d not found: ID does not exist"
Oct 30 00:29:19.799829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.799759    4441 scope.go:122] "RemoveContainer" containerID="58987af5c64e3fa80dd4753614ce45bfded62887cdb075f2e662b7b95bf94211"
Oct 30 00:29:19.800054 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:29:19.800009    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58987af5c64e3fa80dd4753614ce45bfded62887cdb075f2e662b7b95bf94211\": container with ID starting with 58987af5c64e3fa80dd4753614ce45bfded62887cdb075f2e662b7b95bf94211 not found: ID does not exist" containerID="58987af5c64e3fa80dd4753614ce45bfded62887cdb075f2e662b7b95bf94211"
Oct 30 00:29:19.800140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.800068    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58987af5c64e3fa80dd4753614ce45bfded62887cdb075f2e662b7b95bf94211"} err="failed to get container status \"58987af5c64e3fa80dd4753614ce45bfded62887cdb075f2e662b7b95bf94211\": rpc error: code = NotFound desc = could not find container \"58987af5c64e3fa80dd4753614ce45bfded62887cdb075f2e662b7b95bf94211\": container with ID starting with 58987af5c64e3fa80dd4753614ce45bfded62887cdb075f2e662b7b95bf94211 not found: ID does not exist"
Oct 30 00:29:19.824126 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.824100    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-certs\") pod \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") "
Oct 30 00:29:19.824285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.824170    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-data\") pod \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") "
Oct 30 00:29:19.824285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.824214    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-token\" (UniqueName: \"kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-token\") pod \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") "
Oct 30 00:29:19.824408 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.824317    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istiod-ca-cert\") pod \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") "
Oct 30 00:29:19.824408 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.824334    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-certs" pod "4a972f67-aa0b-4444-9b0a-9662f67e6cac" (UID: "4a972f67-aa0b-4444-9b0a-9662f67e6cac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:29:19.824508 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.824458    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-data" pod "4a972f67-aa0b-4444-9b0a-9662f67e6cac" (UID: "4a972f67-aa0b-4444-9b0a-9662f67e6cac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:29:19.824508 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.824500    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-credential-socket\") pod \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") "
Oct 30 00:29:19.824611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.824592    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-socket\") pod \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") "
Oct 30 00:29:19.824668 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.824628    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-kube-api-access-xnqlg\" (UniqueName: \"kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-kube-api-access-xnqlg\") pod \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") "
Oct 30 00:29:19.824668 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.824643    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istiod-ca-cert" pod "4a972f67-aa0b-4444-9b0a-9662f67e6cac" (UID: "4a972f67-aa0b-4444-9b0a-9662f67e6cac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:29:19.824769 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.824680    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-podinfo\") pod \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") "
Oct 30 00:29:19.824769 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.824749    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-envoy\") pod \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\" (UID: \"4a972f67-aa0b-4444-9b0a-9662f67e6cac\") "
Oct 30 00:29:19.824877 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.824768    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-credential-socket" pod "4a972f67-aa0b-4444-9b0a-9662f67e6cac" (UID: "4a972f67-aa0b-4444-9b0a-9662f67e6cac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:29:19.825210 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.825192    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-socket" pod "4a972f67-aa0b-4444-9b0a-9662f67e6cac" (UID: "4a972f67-aa0b-4444-9b0a-9662f67e6cac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:29:19.825441 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.825412    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:19.825441 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.825440    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:19.825662 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.825454    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:19.825662 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.825468    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:19.825662 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.825480    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:19.827335 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.827311    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-podinfo" pod "4a972f67-aa0b-4444-9b0a-9662f67e6cac" (UID: "4a972f67-aa0b-4444-9b0a-9662f67e6cac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:29:19.827435 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.827345    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-kube-api-access-xnqlg" pod "4a972f67-aa0b-4444-9b0a-9662f67e6cac" (UID: "4a972f67-aa0b-4444-9b0a-9662f67e6cac"). InnerVolumeSpecName "kube-api-access-xnqlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:29:19.827435 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.827381    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-token" pod "4a972f67-aa0b-4444-9b0a-9662f67e6cac" (UID: "4a972f67-aa0b-4444-9b0a-9662f67e6cac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:29:19.827571 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.827557    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-envoy" pod "4a972f67-aa0b-4444-9b0a-9662f67e6cac" (UID: "4a972f67-aa0b-4444-9b0a-9662f67e6cac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:29:19.927159 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.927133    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnqlg\" (UniqueName: \"kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-kube-api-access-xnqlg\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:19.927159 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.927158    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:19.927302 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.927169    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:19.927302 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:19.927177    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a972f67-aa0b-4444-9b0a-9662f67e6cac-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:20.119374 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:20.119346    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"]
Oct 30 00:29:20.123042 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:20.123003    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-bb16a8-openshift-default-58d447869-swcr2"]
Oct 30 00:29:20.611033 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:20.610990    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" path="/var/lib/kubelet/pods/4a972f67-aa0b-4444-9b0a-9662f67e6cac/volumes"
Oct 30 00:29:22.680019 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:22.679368    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"]
Oct 30 00:29:22.680717 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:22.680266    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p" podUID="aa997c6d-133a-4980-b6b9-aa5f2977af78" containerName="maas-api" containerID="cri-o://bb6e7c48f9012d19770084ac71411822a611814260648b5de600e045e317bca4" gracePeriod=30
Oct 30 00:29:22.782687 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:22.782656    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"]
Oct 30 00:29:22.873071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:22.871018    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p" podUID="aa997c6d-133a-4980-b6b9-aa5f2977af78" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.136:8443/health\": dial tcp 10.133.0.136:8443: connect: connection refused"
Oct 30 00:29:23.144480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.144138    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:29:23.307092 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.297129    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-maas-api-tls\") pod \"aa997c6d-133a-4980-b6b9-aa5f2977af78\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") "
Oct 30 00:29:23.307092 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.297199    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa997c6d-133a-4980-b6b9-aa5f2977af78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa997c6d-133a-4980-b6b9-aa5f2977af78-openshift-service-ca\") pod \"aa997c6d-133a-4980-b6b9-aa5f2977af78\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") "
Oct 30 00:29:23.307092 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.297224    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-metrics-certs\") pod \"aa997c6d-133a-4980-b6b9-aa5f2977af78\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") "
Oct 30 00:29:23.307092 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.297388    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa997c6d-133a-4980-b6b9-aa5f2977af78-kube-api-access-jv6fj\" (UniqueName: \"kubernetes.io/projected/aa997c6d-133a-4980-b6b9-aa5f2977af78-kube-api-access-jv6fj\") pod \"aa997c6d-133a-4980-b6b9-aa5f2977af78\" (UID: \"aa997c6d-133a-4980-b6b9-aa5f2977af78\") "
Oct 30 00:29:23.307092 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.299283    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa997c6d-133a-4980-b6b9-aa5f2977af78-openshift-service-ca" pod "aa997c6d-133a-4980-b6b9-aa5f2977af78" (UID: "aa997c6d-133a-4980-b6b9-aa5f2977af78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:29:23.324047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.321599    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-maas-api-tls" pod "aa997c6d-133a-4980-b6b9-aa5f2977af78" (UID: "aa997c6d-133a-4980-b6b9-aa5f2977af78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:29:23.338358 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.335612    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-metrics-certs" pod "aa997c6d-133a-4980-b6b9-aa5f2977af78" (UID: "aa997c6d-133a-4980-b6b9-aa5f2977af78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:29:23.341009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.339347    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa997c6d-133a-4980-b6b9-aa5f2977af78-kube-api-access-jv6fj" pod "aa997c6d-133a-4980-b6b9-aa5f2977af78" (UID: "aa997c6d-133a-4980-b6b9-aa5f2977af78"). InnerVolumeSpecName "kube-api-access-jv6fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:29:23.399775 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.399740    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:23.399775 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.399780    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa997c6d-133a-4980-b6b9-aa5f2977af78-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:23.399991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.399795    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa997c6d-133a-4980-b6b9-aa5f2977af78-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:23.399991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.399819    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv6fj\" (UniqueName: \"kubernetes.io/projected/aa997c6d-133a-4980-b6b9-aa5f2977af78-kube-api-access-jv6fj\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:23.809996 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.809954    4441 generic.go:356] "Generic (PLEG): container finished" podID="aa997c6d-133a-4980-b6b9-aa5f2977af78" containerID="bb6e7c48f9012d19770084ac71411822a611814260648b5de600e045e317bca4" exitCode=0
Oct 30 00:29:23.810407 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.810166    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p" event={"ID":"aa997c6d-133a-4980-b6b9-aa5f2977af78","Type":"ContainerDied","Data":"bb6e7c48f9012d19770084ac71411822a611814260648b5de600e045e317bca4"}
Oct 30 00:29:23.810407 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.810220    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p" event={"ID":"aa997c6d-133a-4980-b6b9-aa5f2977af78","Type":"ContainerDied","Data":"aed78ccc6f031ac7f705efd3de1e123c6f19ed3108ab33c6ea1d06c4d4d6d65b"}
Oct 30 00:29:23.810407 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.810250    4441 scope.go:122] "RemoveContainer" containerID="bb6e7c48f9012d19770084ac71411822a611814260648b5de600e045e317bca4"
Oct 30 00:29:23.810563 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.810410    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"
Oct 30 00:29:23.824177 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.824152    4441 scope.go:122] "RemoveContainer" containerID="bb6e7c48f9012d19770084ac71411822a611814260648b5de600e045e317bca4"
Oct 30 00:29:23.824651 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:29:23.824619    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6e7c48f9012d19770084ac71411822a611814260648b5de600e045e317bca4\": container with ID starting with bb6e7c48f9012d19770084ac71411822a611814260648b5de600e045e317bca4 not found: ID does not exist" containerID="bb6e7c48f9012d19770084ac71411822a611814260648b5de600e045e317bca4"
Oct 30 00:29:23.824738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.824662    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6e7c48f9012d19770084ac71411822a611814260648b5de600e045e317bca4"} err="failed to get container status \"bb6e7c48f9012d19770084ac71411822a611814260648b5de600e045e317bca4\": rpc error: code = NotFound desc = could not find container \"bb6e7c48f9012d19770084ac71411822a611814260648b5de600e045e317bca4\": container with ID starting with bb6e7c48f9012d19770084ac71411822a611814260648b5de600e045e317bca4 not found: ID does not exist"
Oct 30 00:29:23.857670 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.857640    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"]
Oct 30 00:29:23.870309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:23.870271    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4500d5-85f594cd76-knf8p"]
Oct 30 00:29:24.610992 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:24.610959    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa997c6d-133a-4980-b6b9-aa5f2977af78" path="/var/lib/kubelet/pods/aa997c6d-133a-4980-b6b9-aa5f2977af78/volumes"
Oct 30 00:29:27.783991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:27.783931    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4" podUID="768f98d2-4192-4df7-8b73-19c8b39029b4" containerName="payload-processing" containerID="cri-o://2783553b201e9f8fe3a5126f367fa5a8edf50a051c67663a35342786ee959f43" gracePeriod=25
Oct 30 00:29:28.101965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.101928    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"
Oct 30 00:29:28.186704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.186675    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768f98d2-4192-4df7-8b73-19c8b39029b4-kube-api-access-6k587\" (UniqueName: \"kubernetes.io/projected/768f98d2-4192-4df7-8b73-19c8b39029b4-kube-api-access-6k587\") pod \"768f98d2-4192-4df7-8b73-19c8b39029b4\" (UID: \"768f98d2-4192-4df7-8b73-19c8b39029b4\") "
Oct 30 00:29:28.186900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.186873    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/768f98d2-4192-4df7-8b73-19c8b39029b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768f98d2-4192-4df7-8b73-19c8b39029b4-plugins-config-volume\") pod \"768f98d2-4192-4df7-8b73-19c8b39029b4\" (UID: \"768f98d2-4192-4df7-8b73-19c8b39029b4\") "
Oct 30 00:29:28.187258 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.187230    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768f98d2-4192-4df7-8b73-19c8b39029b4-plugins-config-volume" pod "768f98d2-4192-4df7-8b73-19c8b39029b4" (UID: "768f98d2-4192-4df7-8b73-19c8b39029b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:29:28.187607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.187584    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768f98d2-4192-4df7-8b73-19c8b39029b4-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:28.190608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.190581    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768f98d2-4192-4df7-8b73-19c8b39029b4-kube-api-access-6k587" pod "768f98d2-4192-4df7-8b73-19c8b39029b4" (UID: "768f98d2-4192-4df7-8b73-19c8b39029b4"). InnerVolumeSpecName "kube-api-access-6k587". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:29:28.288934 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.288905    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k587\" (UniqueName: \"kubernetes.io/projected/768f98d2-4192-4df7-8b73-19c8b39029b4-kube-api-access-6k587\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:28.851778 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.851744    4441 generic.go:356] "Generic (PLEG): container finished" podID="768f98d2-4192-4df7-8b73-19c8b39029b4" containerID="2783553b201e9f8fe3a5126f367fa5a8edf50a051c67663a35342786ee959f43" exitCode=0
Oct 30 00:29:28.852264 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.851790    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"
Oct 30 00:29:28.852264 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.851839    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4" event={"ID":"768f98d2-4192-4df7-8b73-19c8b39029b4","Type":"ContainerDied","Data":"2783553b201e9f8fe3a5126f367fa5a8edf50a051c67663a35342786ee959f43"}
Oct 30 00:29:28.852264 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.851891    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4" event={"ID":"768f98d2-4192-4df7-8b73-19c8b39029b4","Type":"ContainerDied","Data":"10772de9546509f287e3ec89a4a59f04a09fe9385b6426210cecedc0b78ceb7c"}
Oct 30 00:29:28.852264 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.851917    4441 scope.go:122] "RemoveContainer" containerID="2783553b201e9f8fe3a5126f367fa5a8edf50a051c67663a35342786ee959f43"
Oct 30 00:29:28.862540 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.862508    4441 scope.go:122] "RemoveContainer" containerID="2783553b201e9f8fe3a5126f367fa5a8edf50a051c67663a35342786ee959f43"
Oct 30 00:29:28.862832 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:29:28.862805    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2783553b201e9f8fe3a5126f367fa5a8edf50a051c67663a35342786ee959f43\": container with ID starting with 2783553b201e9f8fe3a5126f367fa5a8edf50a051c67663a35342786ee959f43 not found: ID does not exist" containerID="2783553b201e9f8fe3a5126f367fa5a8edf50a051c67663a35342786ee959f43"
Oct 30 00:29:28.862936 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.862844    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2783553b201e9f8fe3a5126f367fa5a8edf50a051c67663a35342786ee959f43"} err="failed to get container status \"2783553b201e9f8fe3a5126f367fa5a8edf50a051c67663a35342786ee959f43\": rpc error: code = NotFound desc = could not find container \"2783553b201e9f8fe3a5126f367fa5a8edf50a051c67663a35342786ee959f43\": container with ID starting with 2783553b201e9f8fe3a5126f367fa5a8edf50a051c67663a35342786ee959f43 not found: ID does not exist"
Oct 30 00:29:28.900104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.900041    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"]
Oct 30 00:29:28.909710 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:28.909672    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-43dc72-db8996694-scsv4"]
Oct 30 00:29:30.612898 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:30.612851    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="768f98d2-4192-4df7-8b73-19c8b39029b4" path="/var/lib/kubelet/pods/768f98d2-4192-4df7-8b73-19c8b39029b4/volumes"
Oct 30 00:29:32.292909 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.292870    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"]
Oct 30 00:29:32.293790 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.293754    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87" podUID="fe107d75-925a-4bf8-99db-c497e80dd4b1" containerName="maas-api" containerID="cri-o://0fc40917dc66aae9eabd7f0cda5024e9aee4bb82372ecf21ef395c73292502d5" gracePeriod=30
Oct 30 00:29:32.422460 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.422431    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"]
Oct 30 00:29:32.422961 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.422645    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j" podUID="1c7f7538-6f8f-4605-bac8-a0dfde5e4017" containerName="payload-processing" containerID="cri-o://1d2343552e3befbf91cd86635296510feba80dbb984e25efb749dbb810210d21" gracePeriod=30
Oct 30 00:29:32.700967 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.700619    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:29:32.796050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.784581    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe107d75-925a-4bf8-99db-c497e80dd4b1-kube-api-access-jjwtz\" (UniqueName: \"kubernetes.io/projected/fe107d75-925a-4bf8-99db-c497e80dd4b1-kube-api-access-jjwtz\") pod \"fe107d75-925a-4bf8-99db-c497e80dd4b1\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") "
Oct 30 00:29:32.796050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.784667    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-metrics-certs\") pod \"fe107d75-925a-4bf8-99db-c497e80dd4b1\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") "
Oct 30 00:29:32.796050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.784871    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe107d75-925a-4bf8-99db-c497e80dd4b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe107d75-925a-4bf8-99db-c497e80dd4b1-openshift-service-ca\") pod \"fe107d75-925a-4bf8-99db-c497e80dd4b1\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") "
Oct 30 00:29:32.796050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.784980    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-maas-api-tls\") pod \"fe107d75-925a-4bf8-99db-c497e80dd4b1\" (UID: \"fe107d75-925a-4bf8-99db-c497e80dd4b1\") "
Oct 30 00:29:32.796050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.787317    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe107d75-925a-4bf8-99db-c497e80dd4b1-openshift-service-ca" pod "fe107d75-925a-4bf8-99db-c497e80dd4b1" (UID: "fe107d75-925a-4bf8-99db-c497e80dd4b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:29:32.806495 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.803513    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-maas-api-tls" pod "fe107d75-925a-4bf8-99db-c497e80dd4b1" (UID: "fe107d75-925a-4bf8-99db-c497e80dd4b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:29:32.806495 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.803669    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-metrics-certs" pod "fe107d75-925a-4bf8-99db-c497e80dd4b1" (UID: "fe107d75-925a-4bf8-99db-c497e80dd4b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:29:32.826378 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.822095    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe107d75-925a-4bf8-99db-c497e80dd4b1-kube-api-access-jjwtz" pod "fe107d75-925a-4bf8-99db-c497e80dd4b1" (UID: "fe107d75-925a-4bf8-99db-c497e80dd4b1"). InnerVolumeSpecName "kube-api-access-jjwtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:29:32.887217 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.887185    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjwtz\" (UniqueName: \"kubernetes.io/projected/fe107d75-925a-4bf8-99db-c497e80dd4b1-kube-api-access-jjwtz\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:32.887217 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.887219    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:32.887442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.887234    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe107d75-925a-4bf8-99db-c497e80dd4b1-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:32.887442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.887249    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe107d75-925a-4bf8-99db-c497e80dd4b1-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:32.890430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.890231    4441 generic.go:356] "Generic (PLEG): container finished" podID="fe107d75-925a-4bf8-99db-c497e80dd4b1" containerID="0fc40917dc66aae9eabd7f0cda5024e9aee4bb82372ecf21ef395c73292502d5" exitCode=0
Oct 30 00:29:32.890430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.890257    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87" event={"ID":"fe107d75-925a-4bf8-99db-c497e80dd4b1","Type":"ContainerDied","Data":"0fc40917dc66aae9eabd7f0cda5024e9aee4bb82372ecf21ef395c73292502d5"}
Oct 30 00:29:32.890430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.890273    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"
Oct 30 00:29:32.890430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.890324    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87" event={"ID":"fe107d75-925a-4bf8-99db-c497e80dd4b1","Type":"ContainerDied","Data":"66f3c9663102d149769277bfbe57cf2c474f3701d6b7d8cb293939a52413658d"}
Oct 30 00:29:32.890430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.890348    4441 scope.go:122] "RemoveContainer" containerID="0fc40917dc66aae9eabd7f0cda5024e9aee4bb82372ecf21ef395c73292502d5"
Oct 30 00:29:32.892423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.892397    4441 generic.go:356] "Generic (PLEG): container finished" podID="1c7f7538-6f8f-4605-bac8-a0dfde5e4017" containerID="1d2343552e3befbf91cd86635296510feba80dbb984e25efb749dbb810210d21" exitCode=0
Oct 30 00:29:32.892522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.892482    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j" event={"ID":"1c7f7538-6f8f-4605-bac8-a0dfde5e4017","Type":"ContainerDied","Data":"1d2343552e3befbf91cd86635296510feba80dbb984e25efb749dbb810210d21"}
Oct 30 00:29:32.894413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.894392    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"
Oct 30 00:29:32.905446 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.905431    4441 scope.go:122] "RemoveContainer" containerID="0fc40917dc66aae9eabd7f0cda5024e9aee4bb82372ecf21ef395c73292502d5"
Oct 30 00:29:32.905775 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:29:32.905742    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc40917dc66aae9eabd7f0cda5024e9aee4bb82372ecf21ef395c73292502d5\": container with ID starting with 0fc40917dc66aae9eabd7f0cda5024e9aee4bb82372ecf21ef395c73292502d5 not found: ID does not exist" containerID="0fc40917dc66aae9eabd7f0cda5024e9aee4bb82372ecf21ef395c73292502d5"
Oct 30 00:29:32.905882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:32.905782    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc40917dc66aae9eabd7f0cda5024e9aee4bb82372ecf21ef395c73292502d5"} err="failed to get container status \"0fc40917dc66aae9eabd7f0cda5024e9aee4bb82372ecf21ef395c73292502d5\": rpc error: code = NotFound desc = could not find container \"0fc40917dc66aae9eabd7f0cda5024e9aee4bb82372ecf21ef395c73292502d5\": container with ID starting with 0fc40917dc66aae9eabd7f0cda5024e9aee4bb82372ecf21ef395c73292502d5 not found: ID does not exist"
Oct 30 00:29:33.015048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:33.002716    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-kube-api-access-k9v65\" (UniqueName: \"kubernetes.io/projected/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-kube-api-access-k9v65\") pod \"1c7f7538-6f8f-4605-bac8-a0dfde5e4017\" (UID: \"1c7f7538-6f8f-4605-bac8-a0dfde5e4017\") "
Oct 30 00:29:33.015048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:33.003632    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-plugins-config-volume\") pod \"1c7f7538-6f8f-4605-bac8-a0dfde5e4017\" (UID: \"1c7f7538-6f8f-4605-bac8-a0dfde5e4017\") "
Oct 30 00:29:33.015048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:33.012101    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-plugins-config-volume" pod "1c7f7538-6f8f-4605-bac8-a0dfde5e4017" (UID: "1c7f7538-6f8f-4605-bac8-a0dfde5e4017"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:29:33.015048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:33.013937    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:33.029263 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:33.029213    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"]
Oct 30 00:29:33.029829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:33.029796    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-kube-api-access-k9v65" pod "1c7f7538-6f8f-4605-bac8-a0dfde5e4017" (UID: "1c7f7538-6f8f-4605-bac8-a0dfde5e4017"). InnerVolumeSpecName "kube-api-access-k9v65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:29:33.034753 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:33.034720    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a15230-5fc4b68cf4-gzr87"]
Oct 30 00:29:33.122047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:33.118173    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9v65\" (UniqueName: \"kubernetes.io/projected/1c7f7538-6f8f-4605-bac8-a0dfde5e4017-kube-api-access-k9v65\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:29:33.903427 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:29:33.903394    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:29:33.903915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:33.903747    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"
Oct 30 00:29:33.903915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:33.903746    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j" event={"ID":"1c7f7538-6f8f-4605-bac8-a0dfde5e4017","Type":"ContainerDied","Data":"153302d7e25110aaf6c422ecf4ba47585c68dfdb6a586a4f9a79bd7312273bf4"}
Oct 30 00:29:33.903915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:33.903867    4441 scope.go:122] "RemoveContainer" containerID="1d2343552e3befbf91cd86635296510feba80dbb984e25efb749dbb810210d21"
Oct 30 00:29:33.949363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:33.949331    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"]
Oct 30 00:29:33.960399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:33.960366    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a15230-68b76d95d8-lp69j"]
Oct 30 00:29:34.627270 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:34.627236    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7f7538-6f8f-4605-bac8-a0dfde5e4017" path="/var/lib/kubelet/pods/1c7f7538-6f8f-4605-bac8-a0dfde5e4017/volumes"
Oct 30 00:29:34.628074 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:34.627837    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe107d75-925a-4bf8-99db-c497e80dd4b1" path="/var/lib/kubelet/pods/fe107d75-925a-4bf8-99db-c497e80dd4b1/volumes"
Oct 30 00:29:49.756736 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:49.756700    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:29:53.172564 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:53.172525    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:29:53.200062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:53.197467    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:29:53.284215 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:53.284165    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:29:53.304201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:53.304106    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:29:53.399177 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:53.399132    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:29:53.429050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:53.399923    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:29:53.458060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:53.451107    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:29:53.486296 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:53.486247    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:29:53.490050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:53.486598    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:29:53.764462 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:53.762606    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:29:53.876197 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:53.868329    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:29:56.857533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:56.857499    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:29:56.863879 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:56.863860    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:29:58.512518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:58.512489    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw_e8480250-bbbd-44f3-a690-b5d2cb97efda/payload-processing/0.log"
Oct 30 00:29:59.644515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:59.644476    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"]
Oct 30 00:29:59.647566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:29:59.644737    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td" podUID="18454967-0713-4eb4-93a6-7806304ea164" containerName="maas-api" containerID="cri-o://60348bdfe9df09a69d685179992bf240664ab0f9e5b64205840644f1f0cfb1a8" gracePeriod=30
Oct 30 00:30:00.029699 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.029668    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:30:00.157151 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157113    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"]
Oct 30 00:30:00.157873 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157850    4441 cpu_manager.go:414] "containerMap: removing container" podUID="4bb99ba7-cf69-456b-8384-57ef12a418be" containerName="payload-processing"
Oct 30 00:30:00.157873 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157871    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb99ba7-cf69-456b-8384-57ef12a418be" containerName="payload-processing"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157882    4441 cpu_manager.go:414] "containerMap: removing container" podUID="9a5b0544-84cb-494c-a40d-7b23774cc079" containerName="istio-proxy"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157887    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5b0544-84cb-494c-a40d-7b23774cc079" containerName="istio-proxy"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157894    4441 cpu_manager.go:414] "containerMap: removing container" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157899    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157907    4441 cpu_manager.go:414] "containerMap: removing container" podUID="768f98d2-4192-4df7-8b73-19c8b39029b4" containerName="payload-processing"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157912    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="768f98d2-4192-4df7-8b73-19c8b39029b4" containerName="payload-processing"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157940    4441 cpu_manager.go:414] "containerMap: removing container" podUID="089e7957-fa4a-4bf4-b457-28af567838fd" containerName="payload-processing"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157944    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089e7957-fa4a-4bf4-b457-28af567838fd" containerName="payload-processing"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157957    4441 cpu_manager.go:414] "containerMap: removing container" podUID="663f591e-ad15-44c2-9983-5464126875c2" containerName="payload-pre-processing"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157963    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663f591e-ad15-44c2-9983-5464126875c2" containerName="payload-pre-processing"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157975    4441 cpu_manager.go:414] "containerMap: removing container" podUID="fe107d75-925a-4bf8-99db-c497e80dd4b1" containerName="maas-api"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157982    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe107d75-925a-4bf8-99db-c497e80dd4b1" containerName="maas-api"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.157991    4441 cpu_manager.go:414] "containerMap: removing container" podUID="1f51de93-61c4-40c5-808f-347d21189d3d" containerName="payload-pre-processing"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158000    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f51de93-61c4-40c5-808f-347d21189d3d" containerName="payload-pre-processing"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158010    4441 cpu_manager.go:414] "containerMap: removing container" podUID="71a1c040-83ea-4f04-b8a3-e3f3c59937d9" containerName="payload-processing"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158016    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a1c040-83ea-4f04-b8a3-e3f3c59937d9" containerName="payload-processing"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158086    4441 cpu_manager.go:414] "containerMap: removing container" podUID="aa997c6d-133a-4980-b6b9-aa5f2977af78" containerName="maas-api"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158094    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa997c6d-133a-4980-b6b9-aa5f2977af78" containerName="maas-api"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158106    4441 cpu_manager.go:414] "containerMap: removing container" podUID="18454967-0713-4eb4-93a6-7806304ea164" containerName="maas-api"
Oct 30 00:30:00.158112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158112    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18454967-0713-4eb4-93a6-7806304ea164" containerName="maas-api"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158129    4441 cpu_manager.go:414] "containerMap: removing container" podUID="c554ae97-8f0b-4bf9-bf77-6ac3bfb42575" containerName="payload-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158138    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c554ae97-8f0b-4bf9-bf77-6ac3bfb42575" containerName="payload-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158151    4441 cpu_manager.go:414] "containerMap: removing container" podUID="1c7f7538-6f8f-4605-bac8-a0dfde5e4017" containerName="payload-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158158    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7f7538-6f8f-4605-bac8-a0dfde5e4017" containerName="payload-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158182    4441 cpu_manager.go:414] "containerMap: removing container" podUID="c554ae97-8f0b-4bf9-bf77-6ac3bfb42575" containerName="payload-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158190    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c554ae97-8f0b-4bf9-bf77-6ac3bfb42575" containerName="payload-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158427    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="089e7957-fa4a-4bf4-b457-28af567838fd" containerName="payload-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158457    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="663f591e-ad15-44c2-9983-5464126875c2" containerName="payload-pre-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158470    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe107d75-925a-4bf8-99db-c497e80dd4b1" containerName="maas-api"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158486    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f51de93-61c4-40c5-808f-347d21189d3d" containerName="payload-pre-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158500    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a1c040-83ea-4f04-b8a3-e3f3c59937d9" containerName="payload-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158517    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa997c6d-133a-4980-b6b9-aa5f2977af78" containerName="maas-api"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158533    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="18454967-0713-4eb4-93a6-7806304ea164" containerName="maas-api"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158542    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c554ae97-8f0b-4bf9-bf77-6ac3bfb42575" containerName="payload-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158557    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7f7538-6f8f-4605-bac8-a0dfde5e4017" containerName="payload-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158577    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c554ae97-8f0b-4bf9-bf77-6ac3bfb42575" containerName="payload-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158590    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158601    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb99ba7-cf69-456b-8384-57ef12a418be" containerName="payload-processing"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158610    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5b0544-84cb-494c-a40d-7b23774cc079" containerName="istio-proxy"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158621    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy"
Oct 30 00:30:00.159140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.158628    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="768f98d2-4192-4df7-8b73-19c8b39029b4" containerName="payload-processing"
Oct 30 00:30:00.165077 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.162551    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"
Oct 30 00:30:00.175909 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.175850    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-metrics-certs\") pod \"18454967-0713-4eb4-93a6-7806304ea164\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") "
Oct 30 00:30:00.176131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.176110    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-maas-api-tls\") pod \"18454967-0713-4eb4-93a6-7806304ea164\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") "
Oct 30 00:30:00.185309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.176208    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18454967-0713-4eb4-93a6-7806304ea164-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18454967-0713-4eb4-93a6-7806304ea164-openshift-service-ca\") pod \"18454967-0713-4eb4-93a6-7806304ea164\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") "
Oct 30 00:30:00.185309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.176262    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18454967-0713-4eb4-93a6-7806304ea164-kube-api-access-k75st\" (UniqueName: \"kubernetes.io/projected/18454967-0713-4eb4-93a6-7806304ea164-kube-api-access-k75st\") pod \"18454967-0713-4eb4-93a6-7806304ea164\" (UID: \"18454967-0713-4eb4-93a6-7806304ea164\") "
Oct 30 00:30:00.185309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.180271    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18454967-0713-4eb4-93a6-7806304ea164-openshift-service-ca" pod "18454967-0713-4eb4-93a6-7806304ea164" (UID: "18454967-0713-4eb4-93a6-7806304ea164"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:30:00.185309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.180463    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-metrics-certs" pod "18454967-0713-4eb4-93a6-7806304ea164" (UID: "18454967-0713-4eb4-93a6-7806304ea164"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:30:00.185309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.182844    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\""
Oct 30 00:30:00.185309 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.180013    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18454967-0713-4eb4-93a6-7806304ea164-kube-api-access-k75st" pod "18454967-0713-4eb4-93a6-7806304ea164" (UID: "18454967-0713-4eb4-93a6-7806304ea164"). InnerVolumeSpecName "kube-api-access-k75st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:30:00.194067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.193410    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"]
Oct 30 00:30:00.196720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.194571    4441 cpu_manager.go:414] "containerMap: removing container" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy"
Oct 30 00:30:00.196720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.194593    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a972f67-aa0b-4444-9b0a-9662f67e6cac" containerName="istio-proxy"
Oct 30 00:30:00.196720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.195128    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-maas-api-tls" pod "18454967-0713-4eb4-93a6-7806304ea164" (UID: "18454967-0713-4eb4-93a6-7806304ea164"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:30:00.231686 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.231404    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"
Oct 30 00:30:00.233714 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.231319    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"]
Oct 30 00:30:00.262049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.261249    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"]
Oct 30 00:30:00.262049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.261311    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"]
Oct 30 00:30:00.270788 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.270750    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"
Oct 30 00:30:00.271917 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.271793    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"
Oct 30 00:30:00.280468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.278339    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cfdc37-5f4d-4bcc-a667-da57a16f8301-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw\" (UID: \"83cfdc37-5f4d-4bcc-a667-da57a16f8301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"
Oct 30 00:30:00.280468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.278452    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5xj\" (UniqueName: \"kubernetes.io/projected/83cfdc37-5f4d-4bcc-a667-da57a16f8301-kube-api-access-gg5xj\") pod \"maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw\" (UID: \"83cfdc37-5f4d-4bcc-a667-da57a16f8301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"
Oct 30 00:30:00.280468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.278833    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:00.280468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.278854    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18454967-0713-4eb4-93a6-7806304ea164-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:00.280468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.280258    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k75st\" (UniqueName: \"kubernetes.io/projected/18454967-0713-4eb4-93a6-7806304ea164-kube-api-access-k75st\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:00.280468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.280326    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18454967-0713-4eb4-93a6-7806304ea164-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:00.282080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.281985    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"]
Oct 30 00:30:00.418817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.323212    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"]
Oct 30 00:30:00.418817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.385575    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cfdc37-5f4d-4bcc-a667-da57a16f8301-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw\" (UID: \"83cfdc37-5f4d-4bcc-a667-da57a16f8301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"
Oct 30 00:30:00.418817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.385679    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5xj\" (UniqueName: \"kubernetes.io/projected/83cfdc37-5f4d-4bcc-a667-da57a16f8301-kube-api-access-gg5xj\") pod \"maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw\" (UID: \"83cfdc37-5f4d-4bcc-a667-da57a16f8301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"
Oct 30 00:30:00.418817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.385776    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz\" (UID: \"e692d2fb-a7c5-4b9a-9e19-883eb44d4241\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"
Oct 30 00:30:00.418817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.391609    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6zg\" (UniqueName: \"kubernetes.io/projected/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-kube-api-access-xl6zg\") pod \"maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz\" (UID: \"e692d2fb-a7c5-4b9a-9e19-883eb44d4241\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"
Oct 30 00:30:00.418817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.391685    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884\" (UID: \"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"
Oct 30 00:30:00.418817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.391730    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b85a3-27cc-4e62-917e-b524279557f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr\" (UID: \"7e0b85a3-27cc-4e62-917e-b524279557f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"
Oct 30 00:30:00.418817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.391815    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhpbf\" (UniqueName: \"kubernetes.io/projected/7e0b85a3-27cc-4e62-917e-b524279557f5-kube-api-access-hhpbf\") pod \"maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr\" (UID: \"7e0b85a3-27cc-4e62-917e-b524279557f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"
Oct 30 00:30:00.418817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.391902    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cfdc37-5f4d-4bcc-a667-da57a16f8301-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw\" (UID: \"83cfdc37-5f4d-4bcc-a667-da57a16f8301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"
Oct 30 00:30:00.418817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.391915    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns8kk\" (UniqueName: \"kubernetes.io/projected/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-kube-api-access-ns8kk\") pod \"maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884\" (UID: \"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"
Oct 30 00:30:00.418817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.416263    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5xj\" (UniqueName: \"kubernetes.io/projected/83cfdc37-5f4d-4bcc-a667-da57a16f8301-kube-api-access-gg5xj\") pod \"maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw\" (UID: \"83cfdc37-5f4d-4bcc-a667-da57a16f8301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"
Oct 30 00:30:00.429556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.428697    4441 generic.go:356] "Generic (PLEG): container finished" podID="18454967-0713-4eb4-93a6-7806304ea164" containerID="60348bdfe9df09a69d685179992bf240664ab0f9e5b64205840644f1f0cfb1a8" exitCode=0
Oct 30 00:30:00.429556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.428832    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td" event={"ID":"18454967-0713-4eb4-93a6-7806304ea164","Type":"ContainerDied","Data":"60348bdfe9df09a69d685179992bf240664ab0f9e5b64205840644f1f0cfb1a8"}
Oct 30 00:30:00.429556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.428875    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td" event={"ID":"18454967-0713-4eb4-93a6-7806304ea164","Type":"ContainerDied","Data":"0de2882ec04fe995706e6fc874ab940f0816875008b675578e5ae436e93d025d"}
Oct 30 00:30:00.429556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.428907    4441 scope.go:122] "RemoveContainer" containerID="60348bdfe9df09a69d685179992bf240664ab0f9e5b64205840644f1f0cfb1a8"
Oct 30 00:30:00.429556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.429082    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"
Oct 30 00:30:00.471210 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.471178    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"]
Oct 30 00:30:00.483641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.483618    4441 scope.go:122] "RemoveContainer" containerID="60348bdfe9df09a69d685179992bf240664ab0f9e5b64205840644f1f0cfb1a8"
Oct 30 00:30:00.484008 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:00.483981    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60348bdfe9df09a69d685179992bf240664ab0f9e5b64205840644f1f0cfb1a8\": container with ID starting with 60348bdfe9df09a69d685179992bf240664ab0f9e5b64205840644f1f0cfb1a8 not found: ID does not exist" containerID="60348bdfe9df09a69d685179992bf240664ab0f9e5b64205840644f1f0cfb1a8"
Oct 30 00:30:00.484100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.484063    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60348bdfe9df09a69d685179992bf240664ab0f9e5b64205840644f1f0cfb1a8"} err="failed to get container status \"60348bdfe9df09a69d685179992bf240664ab0f9e5b64205840644f1f0cfb1a8\": rpc error: code = NotFound desc = could not find container \"60348bdfe9df09a69d685179992bf240664ab0f9e5b64205840644f1f0cfb1a8\": container with ID starting with 60348bdfe9df09a69d685179992bf240664ab0f9e5b64205840644f1f0cfb1a8 not found: ID does not exist"
Oct 30 00:30:00.486360 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.486338    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"
Oct 30 00:30:00.493402 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.493382    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz\" (UID: \"e692d2fb-a7c5-4b9a-9e19-883eb44d4241\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"
Oct 30 00:30:00.493522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.493407    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6zg\" (UniqueName: \"kubernetes.io/projected/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-kube-api-access-xl6zg\") pod \"maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz\" (UID: \"e692d2fb-a7c5-4b9a-9e19-883eb44d4241\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"
Oct 30 00:30:00.493522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.493484    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884\" (UID: \"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"
Oct 30 00:30:00.493651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.493529    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b85a3-27cc-4e62-917e-b524279557f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr\" (UID: \"7e0b85a3-27cc-4e62-917e-b524279557f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"
Oct 30 00:30:00.493651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.493604    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhpbf\" (UniqueName: \"kubernetes.io/projected/7e0b85a3-27cc-4e62-917e-b524279557f5-kube-api-access-hhpbf\") pod \"maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr\" (UID: \"7e0b85a3-27cc-4e62-917e-b524279557f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"
Oct 30 00:30:00.493755 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.493696    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns8kk\" (UniqueName: \"kubernetes.io/projected/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-kube-api-access-ns8kk\") pod \"maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884\" (UID: \"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"
Oct 30 00:30:00.494549 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.494521    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b85a3-27cc-4e62-917e-b524279557f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr\" (UID: \"7e0b85a3-27cc-4e62-917e-b524279557f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"
Oct 30 00:30:00.495404 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.495381    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz\" (UID: \"e692d2fb-a7c5-4b9a-9e19-883eb44d4241\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"
Oct 30 00:30:00.495839 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.495815    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884\" (UID: \"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"
Oct 30 00:30:00.502251 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.502222    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"]
Oct 30 00:30:00.505874 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.505849    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-43dc72-58bb667f6d-5j9td"]
Oct 30 00:30:00.514197 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.514179    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns8kk\" (UniqueName: \"kubernetes.io/projected/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-kube-api-access-ns8kk\") pod \"maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884\" (UID: \"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"
Oct 30 00:30:00.514280 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.514253    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6zg\" (UniqueName: \"kubernetes.io/projected/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-kube-api-access-xl6zg\") pod \"maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz\" (UID: \"e692d2fb-a7c5-4b9a-9e19-883eb44d4241\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"
Oct 30 00:30:00.514499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.514482    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhpbf\" (UniqueName: \"kubernetes.io/projected/7e0b85a3-27cc-4e62-917e-b524279557f5-kube-api-access-hhpbf\") pod \"maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr\" (UID: \"7e0b85a3-27cc-4e62-917e-b524279557f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"
Oct 30 00:30:00.623414 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.623363    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"
Oct 30 00:30:00.627131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.627103    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18454967-0713-4eb4-93a6-7806304ea164" path="/var/lib/kubelet/pods/18454967-0713-4eb4-93a6-7806304ea164/volumes"
Oct 30 00:30:00.686706 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.686452    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"
Oct 30 00:30:00.711611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.711585    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"
Oct 30 00:30:00.842845 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:00.842810    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"]
Oct 30 00:30:01.010005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:01.009978    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"]
Oct 30 00:30:01.010630 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:30:01.010600    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b85a3_27cc_4e62_917e_b524279557f5.slice/crio-7abfb73f6e6e2054e9564ae73f030b2907200dcc2e597cc05fc9af449ce17b2e WatchSource:0}: Error finding container 7abfb73f6e6e2054e9564ae73f030b2907200dcc2e597cc05fc9af449ce17b2e: Status 404 returned error can't find the container with id 7abfb73f6e6e2054e9564ae73f030b2907200dcc2e597cc05fc9af449ce17b2e
Oct 30 00:30:01.056993 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:01.056960    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"]
Oct 30 00:30:01.058886 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:30:01.058857    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2563bc_4202_4d5c_92ea_15aaf4cd88ec.slice/crio-7d092bbf082da720503667d03fcf68cf33e9a43528e2724dbaf7490e6839271f WatchSource:0}: Error finding container 7d092bbf082da720503667d03fcf68cf33e9a43528e2724dbaf7490e6839271f: Status 404 returned error can't find the container with id 7d092bbf082da720503667d03fcf68cf33e9a43528e2724dbaf7490e6839271f
Oct 30 00:30:01.069492 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:01.069464    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"]
Oct 30 00:30:01.072356 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:30:01.072328    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode692d2fb_a7c5_4b9a_9e19_883eb44d4241.slice/crio-baedd7b51c4a454a255ce96dfe27e35c9f98e218326bd17596bdc8e513065127 WatchSource:0}: Error finding container baedd7b51c4a454a255ce96dfe27e35c9f98e218326bd17596bdc8e513065127: Status 404 returned error can't find the container with id baedd7b51c4a454a255ce96dfe27e35c9f98e218326bd17596bdc8e513065127
Oct 30 00:30:01.440956 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:01.440924    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr" event={"ID":"7e0b85a3-27cc-4e62-917e-b524279557f5","Type":"ContainerStarted","Data":"7abfb73f6e6e2054e9564ae73f030b2907200dcc2e597cc05fc9af449ce17b2e"}
Oct 30 00:30:01.442779 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:01.442751    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884" event={"ID":"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec","Type":"ContainerStarted","Data":"7d092bbf082da720503667d03fcf68cf33e9a43528e2724dbaf7490e6839271f"}
Oct 30 00:30:01.445592 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:01.445568    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw" event={"ID":"83cfdc37-5f4d-4bcc-a667-da57a16f8301","Type":"ContainerStarted","Data":"3908c22a49683f2148fcda8f71823eea8e70ab7750fe240f45a8a1ad443e7582"}
Oct 30 00:30:01.445723 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:01.445617    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw" event={"ID":"83cfdc37-5f4d-4bcc-a667-da57a16f8301","Type":"ContainerStarted","Data":"e6797d0078e760e7b31324fcd1a3517bddfe50cc62926acad7747553ae2b5706"}
Oct 30 00:30:01.447034 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:01.446994    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz" event={"ID":"e692d2fb-a7c5-4b9a-9e19-883eb44d4241","Type":"ContainerStarted","Data":"baedd7b51c4a454a255ce96dfe27e35c9f98e218326bd17596bdc8e513065127"}
Oct 30 00:30:01.481991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:01.481940    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw" podStartSLOduration=1.481926191 podStartE2EDuration="1.481926191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:01.479494793 +0000 UTC m=+1511.004788670" watchObservedRunningTime="2026-10-30 00:30:01.481926191 +0000 UTC m=+1511.007220067"
Oct 30 00:30:02.457825 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:02.457779    4441 generic.go:356] "Generic (PLEG): container finished" podID="7e0b85a3-27cc-4e62-917e-b524279557f5" containerID="bf64cc229df1f326ffa73c1d083234802d55a1021d20be80259f594339475669" exitCode=60
Oct 30 00:30:02.458734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:02.458249    4441 scope.go:122] "RemoveContainer" containerID="bf64cc229df1f326ffa73c1d083234802d55a1021d20be80259f594339475669"
Oct 30 00:30:02.458849 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:02.458813    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr" event={"ID":"7e0b85a3-27cc-4e62-917e-b524279557f5","Type":"ContainerDied","Data":"bf64cc229df1f326ffa73c1d083234802d55a1021d20be80259f594339475669"}
Oct 30 00:30:02.460885 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:02.460860    4441 generic.go:356] "Generic (PLEG): container finished" podID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" containerID="c3253ea64a8721f423f9986ae670e9e5956e9be4adadff6583f5f86fb0744e2a" exitCode=60
Oct 30 00:30:02.460972 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:02.460957    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884" event={"ID":"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec","Type":"ContainerDied","Data":"c3253ea64a8721f423f9986ae670e9e5956e9be4adadff6583f5f86fb0744e2a"}
Oct 30 00:30:02.461388 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:02.461368    4441 scope.go:122] "RemoveContainer" containerID="c3253ea64a8721f423f9986ae670e9e5956e9be4adadff6583f5f86fb0744e2a"
Oct 30 00:30:02.462982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:02.462957    4441 generic.go:356] "Generic (PLEG): container finished" podID="83cfdc37-5f4d-4bcc-a667-da57a16f8301" containerID="3908c22a49683f2148fcda8f71823eea8e70ab7750fe240f45a8a1ad443e7582" exitCode=60
Oct 30 00:30:02.463109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:02.463068    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw" event={"ID":"83cfdc37-5f4d-4bcc-a667-da57a16f8301","Type":"ContainerDied","Data":"3908c22a49683f2148fcda8f71823eea8e70ab7750fe240f45a8a1ad443e7582"}
Oct 30 00:30:02.463402 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:02.463380    4441 scope.go:122] "RemoveContainer" containerID="3908c22a49683f2148fcda8f71823eea8e70ab7750fe240f45a8a1ad443e7582"
Oct 30 00:30:02.465528 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:02.465507    4441 generic.go:356] "Generic (PLEG): container finished" podID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241" containerID="570e96ae679f06310e67926f318b3e29e93ca163f07bf3ce37dd5fe3cb7e34e7" exitCode=60
Oct 30 00:30:02.465621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:02.465563    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz" event={"ID":"e692d2fb-a7c5-4b9a-9e19-883eb44d4241","Type":"ContainerDied","Data":"570e96ae679f06310e67926f318b3e29e93ca163f07bf3ce37dd5fe3cb7e34e7"}
Oct 30 00:30:02.465899 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:02.465883    4441 scope.go:122] "RemoveContainer" containerID="570e96ae679f06310e67926f318b3e29e93ca163f07bf3ce37dd5fe3cb7e34e7"
Oct 30 00:30:03.475990 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:03.475957    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr" event={"ID":"7e0b85a3-27cc-4e62-917e-b524279557f5","Type":"ContainerStarted","Data":"a0c15dd9de2807e912ef28e36ee1ffd36c51d96c6075df89b83a72309e702162"}
Oct 30 00:30:03.477990 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:03.477967    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884" event={"ID":"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec","Type":"ContainerStarted","Data":"2d947df799bee9f5e771277474bc79862002804ba973d0ea37d01656cee401ac"}
Oct 30 00:30:03.479897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:03.479873    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw" event={"ID":"83cfdc37-5f4d-4bcc-a667-da57a16f8301","Type":"ContainerStarted","Data":"840ba9e391fe06c9fde09797b104518184f53cd7ff8cc8b1b6662de4cb0671d4"}
Oct 30 00:30:03.481739 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:03.481714    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz" event={"ID":"e692d2fb-a7c5-4b9a-9e19-883eb44d4241","Type":"ContainerStarted","Data":"49293e5da418d94ab416bb52a47b6682a961a950037220cbf23b176d247fb948"}
Oct 30 00:30:03.513223 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:03.513180    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr" podStartSLOduration=3.513166364 podStartE2EDuration="3.513166364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:03.50970437 +0000 UTC m=+1513.034998271" watchObservedRunningTime="2026-10-30 00:30:03.513166364 +0000 UTC m=+1513.038460241"
Oct 30 00:30:03.544883 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:03.544841    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884" podStartSLOduration=3.544829296 podStartE2EDuration="3.544829296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:03.543766583 +0000 UTC m=+1513.069060460" watchObservedRunningTime="2026-10-30 00:30:03.544829296 +0000 UTC m=+1513.070123173"
Oct 30 00:30:03.580153 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:03.579604    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz" podStartSLOduration=3.579401125 podStartE2EDuration="3.579401125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:03.578562312 +0000 UTC m=+1513.103856190" watchObservedRunningTime="2026-10-30 00:30:03.579401125 +0000 UTC m=+1513.104695027"
Oct 30 00:30:03.914464 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:03.914426    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:30:04.539192 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.539149    4441 generic.go:356] "Generic (PLEG): container finished" podID="7e0b85a3-27cc-4e62-917e-b524279557f5" containerID="a0c15dd9de2807e912ef28e36ee1ffd36c51d96c6075df89b83a72309e702162" exitCode=60
Oct 30 00:30:04.539836 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.539262    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr" event={"ID":"7e0b85a3-27cc-4e62-917e-b524279557f5","Type":"ContainerDied","Data":"a0c15dd9de2807e912ef28e36ee1ffd36c51d96c6075df89b83a72309e702162"}
Oct 30 00:30:04.539836 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.539773    4441 scope.go:122] "RemoveContainer" containerID="bf64cc229df1f326ffa73c1d083234802d55a1021d20be80259f594339475669"
Oct 30 00:30:04.540434 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.540410    4441 scope.go:122] "RemoveContainer" containerID="a0c15dd9de2807e912ef28e36ee1ffd36c51d96c6075df89b83a72309e702162"
Oct 30 00:30:04.540848 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:04.540819    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr_odh-ai-gateway-infra(7e0b85a3-27cc-4e62-917e-b524279557f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr" podUID="7e0b85a3-27cc-4e62-917e-b524279557f5"
Oct 30 00:30:04.543005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.542977    4441 generic.go:356] "Generic (PLEG): container finished" podID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" containerID="2d947df799bee9f5e771277474bc79862002804ba973d0ea37d01656cee401ac" exitCode=60
Oct 30 00:30:04.543148 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.543086    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884" event={"ID":"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec","Type":"ContainerDied","Data":"2d947df799bee9f5e771277474bc79862002804ba973d0ea37d01656cee401ac"}
Oct 30 00:30:04.543453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.543435    4441 scope.go:122] "RemoveContainer" containerID="2d947df799bee9f5e771277474bc79862002804ba973d0ea37d01656cee401ac"
Oct 30 00:30:04.543776 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:04.543757    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884_odh-ai-gateway-infra(6c2563bc-4202-4d5c-92ea-15aaf4cd88ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884" podUID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec"
Oct 30 00:30:04.545762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.545742    4441 generic.go:356] "Generic (PLEG): container finished" podID="83cfdc37-5f4d-4bcc-a667-da57a16f8301" containerID="840ba9e391fe06c9fde09797b104518184f53cd7ff8cc8b1b6662de4cb0671d4" exitCode=60
Oct 30 00:30:04.545905 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.545887    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw" event={"ID":"83cfdc37-5f4d-4bcc-a667-da57a16f8301","Type":"ContainerDied","Data":"840ba9e391fe06c9fde09797b104518184f53cd7ff8cc8b1b6662de4cb0671d4"}
Oct 30 00:30:04.546113 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.546095    4441 scope.go:122] "RemoveContainer" containerID="840ba9e391fe06c9fde09797b104518184f53cd7ff8cc8b1b6662de4cb0671d4"
Oct 30 00:30:04.546470 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:04.546447    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw_odh-ai-gateway-infra(83cfdc37-5f4d-4bcc-a667-da57a16f8301)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw" podUID="83cfdc37-5f4d-4bcc-a667-da57a16f8301"
Oct 30 00:30:04.548657 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.548637    4441 generic.go:356] "Generic (PLEG): container finished" podID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241" containerID="49293e5da418d94ab416bb52a47b6682a961a950037220cbf23b176d247fb948" exitCode=60
Oct 30 00:30:04.548834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.548720    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz" event={"ID":"e692d2fb-a7c5-4b9a-9e19-883eb44d4241","Type":"ContainerDied","Data":"49293e5da418d94ab416bb52a47b6682a961a950037220cbf23b176d247fb948"}
Oct 30 00:30:04.548977 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.548960    4441 scope.go:122] "RemoveContainer" containerID="49293e5da418d94ab416bb52a47b6682a961a950037220cbf23b176d247fb948"
Oct 30 00:30:04.549324 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:04.549303    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz_odh-ai-gateway-infra(e692d2fb-a7c5-4b9a-9e19-883eb44d4241)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz" podUID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241"
Oct 30 00:30:04.554081 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.554063    4441 scope.go:122] "RemoveContainer" containerID="c3253ea64a8721f423f9986ae670e9e5956e9be4adadff6583f5f86fb0744e2a"
Oct 30 00:30:04.569172 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.569147    4441 scope.go:122] "RemoveContainer" containerID="3908c22a49683f2148fcda8f71823eea8e70ab7750fe240f45a8a1ad443e7582"
Oct 30 00:30:04.584682 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:04.584658    4441 scope.go:122] "RemoveContainer" containerID="570e96ae679f06310e67926f318b3e29e93ca163f07bf3ce37dd5fe3cb7e34e7"
Oct 30 00:30:05.560212 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:05.560182    4441 scope.go:122] "RemoveContainer" containerID="2d947df799bee9f5e771277474bc79862002804ba973d0ea37d01656cee401ac"
Oct 30 00:30:05.561002 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:05.560587    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884_odh-ai-gateway-infra(6c2563bc-4202-4d5c-92ea-15aaf4cd88ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884" podUID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec"
Oct 30 00:30:05.562017 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:05.561996    4441 scope.go:122] "RemoveContainer" containerID="840ba9e391fe06c9fde09797b104518184f53cd7ff8cc8b1b6662de4cb0671d4"
Oct 30 00:30:05.562341 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:05.562320    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw_odh-ai-gateway-infra(83cfdc37-5f4d-4bcc-a667-da57a16f8301)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw" podUID="83cfdc37-5f4d-4bcc-a667-da57a16f8301"
Oct 30 00:30:05.563936 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:05.563914    4441 scope.go:122] "RemoveContainer" containerID="49293e5da418d94ab416bb52a47b6682a961a950037220cbf23b176d247fb948"
Oct 30 00:30:05.564310 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:05.564258    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz_odh-ai-gateway-infra(e692d2fb-a7c5-4b9a-9e19-883eb44d4241)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz" podUID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241"
Oct 30 00:30:05.565678 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:05.565660    4441 scope.go:122] "RemoveContainer" containerID="a0c15dd9de2807e912ef28e36ee1ffd36c51d96c6075df89b83a72309e702162"
Oct 30 00:30:05.565920 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:05.565900    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr_odh-ai-gateway-infra(7e0b85a3-27cc-4e62-917e-b524279557f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr" podUID="7e0b85a3-27cc-4e62-917e-b524279557f5"
Oct 30 00:30:15.601842 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:15.601802    4441 scope.go:122] "RemoveContainer" containerID="a0c15dd9de2807e912ef28e36ee1ffd36c51d96c6075df89b83a72309e702162"
Oct 30 00:30:16.057189 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.057153    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"]
Oct 30 00:30:16.061489 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.061469    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"
Oct 30 00:30:16.078341 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.078320    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-1dd8c1\""
Oct 30 00:30:16.087748 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.087728    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-1dd8c1-dockercfg-hdxqs\""
Oct 30 00:30:16.093524 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.093498    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"]
Oct 30 00:30:16.164116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.160998    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5sqs\" (UniqueName: \"kubernetes.io/projected/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-kube-api-access-t5sqs\") pod \"payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg\" (UID: \"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"
Oct 30 00:30:16.164116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.161101    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg\" (UID: \"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"
Oct 30 00:30:16.270579 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.270543    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5sqs\" (UniqueName: \"kubernetes.io/projected/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-kube-api-access-t5sqs\") pod \"payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg\" (UID: \"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"
Oct 30 00:30:16.270908 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.270882    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg\" (UID: \"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"
Oct 30 00:30:16.271858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.271827    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg\" (UID: \"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"
Oct 30 00:30:16.330060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.329762    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5sqs\" (UniqueName: \"kubernetes.io/projected/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-kube-api-access-t5sqs\") pod \"payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg\" (UID: \"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"
Oct 30 00:30:16.378628 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.378595    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"
Oct 30 00:30:16.602454 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.602380    4441 scope.go:122] "RemoveContainer" containerID="49293e5da418d94ab416bb52a47b6682a961a950037220cbf23b176d247fb948"
Oct 30 00:30:16.661285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.661252    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr" event={"ID":"7e0b85a3-27cc-4e62-917e-b524279557f5","Type":"ContainerStarted","Data":"a9546953d3b3548f53639e9a639e8161b777b6db0ea2584e7cecbe369c72482e"}
Oct 30 00:30:16.709757 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:16.709718    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"]
Oct 30 00:30:16.724391 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:30:16.724353    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf06d4e0_8bf5_441f_8e59_dd61fa7785f3.slice/crio-ac3de624db36b948f16b8b5ea6909c53718d9687055f192e2b46369ea1131666 WatchSource:0}: Error finding container ac3de624db36b948f16b8b5ea6909c53718d9687055f192e2b46369ea1131666: Status 404 returned error can't find the container with id ac3de624db36b948f16b8b5ea6909c53718d9687055f192e2b46369ea1131666
Oct 30 00:30:17.601886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.601853    4441 scope.go:122] "RemoveContainer" containerID="840ba9e391fe06c9fde09797b104518184f53cd7ff8cc8b1b6662de4cb0671d4"
Oct 30 00:30:17.602122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.601959    4441 scope.go:122] "RemoveContainer" containerID="2d947df799bee9f5e771277474bc79862002804ba973d0ea37d01656cee401ac"
Oct 30 00:30:17.675094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.675005    4441 generic.go:356] "Generic (PLEG): container finished" podID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241" containerID="aaf3b30fd2b765126a9d73d04876a5c5994848484246c2c28fb31f52b2c45662" exitCode=60
Oct 30 00:30:17.675094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.675058    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz" event={"ID":"e692d2fb-a7c5-4b9a-9e19-883eb44d4241","Type":"ContainerDied","Data":"aaf3b30fd2b765126a9d73d04876a5c5994848484246c2c28fb31f52b2c45662"}
Oct 30 00:30:17.675584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.675125    4441 scope.go:122] "RemoveContainer" containerID="49293e5da418d94ab416bb52a47b6682a961a950037220cbf23b176d247fb948"
Oct 30 00:30:17.675584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.675514    4441 scope.go:122] "RemoveContainer" containerID="aaf3b30fd2b765126a9d73d04876a5c5994848484246c2c28fb31f52b2c45662"
Oct 30 00:30:17.675860 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:17.675836    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz_odh-ai-gateway-infra(e692d2fb-a7c5-4b9a-9e19-883eb44d4241)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz" podUID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241"
Oct 30 00:30:17.677484 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.677451    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg" event={"ID":"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3","Type":"ContainerStarted","Data":"cb21530b471c8ea20acd35bdae90f34e5e80df1b78b636b0bad8f6f9b021f768"}
Oct 30 00:30:17.677604 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.677503    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg" event={"ID":"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3","Type":"ContainerStarted","Data":"ac3de624db36b948f16b8b5ea6909c53718d9687055f192e2b46369ea1131666"}
Oct 30 00:30:17.677604 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.677549    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"
Oct 30 00:30:17.679814 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.679783    4441 generic.go:356] "Generic (PLEG): container finished" podID="7e0b85a3-27cc-4e62-917e-b524279557f5" containerID="a9546953d3b3548f53639e9a639e8161b777b6db0ea2584e7cecbe369c72482e" exitCode=60
Oct 30 00:30:17.679909 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.679848    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr" event={"ID":"7e0b85a3-27cc-4e62-917e-b524279557f5","Type":"ContainerDied","Data":"a9546953d3b3548f53639e9a639e8161b777b6db0ea2584e7cecbe369c72482e"}
Oct 30 00:30:17.680240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.680225    4441 scope.go:122] "RemoveContainer" containerID="a9546953d3b3548f53639e9a639e8161b777b6db0ea2584e7cecbe369c72482e"
Oct 30 00:30:17.680548 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:17.680529    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr_odh-ai-gateway-infra(7e0b85a3-27cc-4e62-917e-b524279557f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr" podUID="7e0b85a3-27cc-4e62-917e-b524279557f5"
Oct 30 00:30:17.693175 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.693158    4441 scope.go:122] "RemoveContainer" containerID="a0c15dd9de2807e912ef28e36ee1ffd36c51d96c6075df89b83a72309e702162"
Oct 30 00:30:17.709013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.708980    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"]
Oct 30 00:30:17.758040 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:17.757968    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg" podStartSLOduration=1.757950619 podStartE2EDuration="1.757950619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:17.751921156 +0000 UTC m=+1527.277215035" watchObservedRunningTime="2026-10-30 00:30:17.757950619 +0000 UTC m=+1527.283244497"
Oct 30 00:30:18.696816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:18.696787    4441 generic.go:356] "Generic (PLEG): container finished" podID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" containerID="6b7fdadb4cc5ce6a0f94b7de193041154700bd9375f440545076aefd6943deb4" exitCode=60
Oct 30 00:30:18.697294 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:18.696868    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884" event={"ID":"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec","Type":"ContainerDied","Data":"6b7fdadb4cc5ce6a0f94b7de193041154700bd9375f440545076aefd6943deb4"}
Oct 30 00:30:18.697294 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:18.696927    4441 scope.go:122] "RemoveContainer" containerID="2d947df799bee9f5e771277474bc79862002804ba973d0ea37d01656cee401ac"
Oct 30 00:30:18.697450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:18.697430    4441 scope.go:122] "RemoveContainer" containerID="6b7fdadb4cc5ce6a0f94b7de193041154700bd9375f440545076aefd6943deb4"
Oct 30 00:30:18.697814 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:18.697789    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884_odh-ai-gateway-infra(6c2563bc-4202-4d5c-92ea-15aaf4cd88ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884" podUID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec"
Oct 30 00:30:18.702921 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:18.702890    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw" event={"ID":"83cfdc37-5f4d-4bcc-a667-da57a16f8301","Type":"ContainerStarted","Data":"ee6ef11289ee8afe84281fa6ea5164225be1dbcb1444a0e9adf41abab6ed7e11"}
Oct 30 00:30:18.724564 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:18.724443    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"]
Oct 30 00:30:18.945981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:18.945643    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"
Oct 30 00:30:19.008445 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.008411    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"
Oct 30 00:30:19.017700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.017670    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0b85a3-27cc-4e62-917e-b524279557f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b85a3-27cc-4e62-917e-b524279557f5-maas-api-service-ca\") pod \"7e0b85a3-27cc-4e62-917e-b524279557f5\" (UID: \"7e0b85a3-27cc-4e62-917e-b524279557f5\") "
Oct 30 00:30:19.018002 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.017954    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0b85a3-27cc-4e62-917e-b524279557f5-kube-api-access-hhpbf\" (UniqueName: \"kubernetes.io/projected/7e0b85a3-27cc-4e62-917e-b524279557f5-kube-api-access-hhpbf\") pod \"7e0b85a3-27cc-4e62-917e-b524279557f5\" (UID: \"7e0b85a3-27cc-4e62-917e-b524279557f5\") "
Oct 30 00:30:19.018479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.018195    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0b85a3-27cc-4e62-917e-b524279557f5-maas-api-service-ca" pod "7e0b85a3-27cc-4e62-917e-b524279557f5" (UID: "7e0b85a3-27cc-4e62-917e-b524279557f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:30:19.018732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.018711    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b85a3-27cc-4e62-917e-b524279557f5-maas-api-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:19.020721 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.020686    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0b85a3-27cc-4e62-917e-b524279557f5-kube-api-access-hhpbf" pod "7e0b85a3-27cc-4e62-917e-b524279557f5" (UID: "7e0b85a3-27cc-4e62-917e-b524279557f5"). InnerVolumeSpecName "kube-api-access-hhpbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:30:19.121211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.119800    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-maas-api-service-ca\") pod \"e692d2fb-a7c5-4b9a-9e19-883eb44d4241\" (UID: \"e692d2fb-a7c5-4b9a-9e19-883eb44d4241\") "
Oct 30 00:30:19.121211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.120162    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-kube-api-access-xl6zg\" (UniqueName: \"kubernetes.io/projected/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-kube-api-access-xl6zg\") pod \"e692d2fb-a7c5-4b9a-9e19-883eb44d4241\" (UID: \"e692d2fb-a7c5-4b9a-9e19-883eb44d4241\") "
Oct 30 00:30:19.121211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.120830    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhpbf\" (UniqueName: \"kubernetes.io/projected/7e0b85a3-27cc-4e62-917e-b524279557f5-kube-api-access-hhpbf\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:19.121897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.121873    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-maas-api-service-ca" pod "e692d2fb-a7c5-4b9a-9e19-883eb44d4241" (UID: "e692d2fb-a7c5-4b9a-9e19-883eb44d4241"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:30:19.123533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.123506    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-kube-api-access-xl6zg" pod "e692d2fb-a7c5-4b9a-9e19-883eb44d4241" (UID: "e692d2fb-a7c5-4b9a-9e19-883eb44d4241"). InnerVolumeSpecName "kube-api-access-xl6zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:30:19.222832 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.222755    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6zg\" (UniqueName: \"kubernetes.io/projected/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-kube-api-access-xl6zg\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:19.222832 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.222783    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e692d2fb-a7c5-4b9a-9e19-883eb44d4241-maas-api-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:19.710505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.710479    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"
Oct 30 00:30:19.710985 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.710513    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz" event={"ID":"e692d2fb-a7c5-4b9a-9e19-883eb44d4241","Type":"ContainerDied","Data":"baedd7b51c4a454a255ce96dfe27e35c9f98e218326bd17596bdc8e513065127"}
Oct 30 00:30:19.710985 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.710576    4441 scope.go:122] "RemoveContainer" containerID="aaf3b30fd2b765126a9d73d04876a5c5994848484246c2c28fb31f52b2c45662"
Oct 30 00:30:19.712231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.712209    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"
Oct 30 00:30:19.712361 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.712250    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr" event={"ID":"7e0b85a3-27cc-4e62-917e-b524279557f5","Type":"ContainerDied","Data":"7abfb73f6e6e2054e9564ae73f030b2907200dcc2e597cc05fc9af449ce17b2e"}
Oct 30 00:30:19.716005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.715983    4441 generic.go:356] "Generic (PLEG): container finished" podID="83cfdc37-5f4d-4bcc-a667-da57a16f8301" containerID="ee6ef11289ee8afe84281fa6ea5164225be1dbcb1444a0e9adf41abab6ed7e11" exitCode=60
Oct 30 00:30:19.716113 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.716045    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw" event={"ID":"83cfdc37-5f4d-4bcc-a667-da57a16f8301","Type":"ContainerDied","Data":"ee6ef11289ee8afe84281fa6ea5164225be1dbcb1444a0e9adf41abab6ed7e11"}
Oct 30 00:30:19.716423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.716406    4441 scope.go:122] "RemoveContainer" containerID="ee6ef11289ee8afe84281fa6ea5164225be1dbcb1444a0e9adf41abab6ed7e11"
Oct 30 00:30:19.717009 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:19.716751    4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw_odh-ai-gateway-infra(83cfdc37-5f4d-4bcc-a667-da57a16f8301)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw" podUID="83cfdc37-5f4d-4bcc-a667-da57a16f8301"
Oct 30 00:30:19.725589 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.725567    4441 scope.go:122] "RemoveContainer" containerID="a9546953d3b3548f53639e9a639e8161b777b6db0ea2584e7cecbe369c72482e"
Oct 30 00:30:19.737750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.737715    4441 scope.go:122] "RemoveContainer" containerID="840ba9e391fe06c9fde09797b104518184f53cd7ff8cc8b1b6662de4cb0671d4"
Oct 30 00:30:19.778567 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.778543    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"]
Oct 30 00:30:19.814892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.814859    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"]
Oct 30 00:30:19.824843 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.824813    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888670-c96pr"]
Oct 30 00:30:19.851948 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.850255    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"]
Oct 30 00:30:19.858798 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.858747    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"]
Oct 30 00:30:19.873051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.872945    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1c718-29888670-qvlsz"]
Oct 30 00:30:19.982358 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:19.982333    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"
Oct 30 00:30:20.036474 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.036440    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-kube-api-access-ns8kk\" (UniqueName: \"kubernetes.io/projected/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-kube-api-access-ns8kk\") pod \"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec\" (UID: \"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec\") "
Oct 30 00:30:20.036733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.036707    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-maas-api-service-ca\") pod \"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec\" (UID: \"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec\") "
Oct 30 00:30:20.038467 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.038440    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-maas-api-service-ca" pod "6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" (UID: "6c2563bc-4202-4d5c-92ea-15aaf4cd88ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:30:20.044178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.044146    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-kube-api-access-ns8kk" pod "6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" (UID: "6c2563bc-4202-4d5c-92ea-15aaf4cd88ec"). InnerVolumeSpecName "kube-api-access-ns8kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:30:20.139343 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.139310    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns8kk\" (UniqueName: \"kubernetes.io/projected/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-kube-api-access-ns8kk\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:20.139343 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.139337    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec-maas-api-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:20.610834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.610803    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0b85a3-27cc-4e62-917e-b524279557f5" path="/var/lib/kubelet/pods/7e0b85a3-27cc-4e62-917e-b524279557f5/volumes"
Oct 30 00:30:20.611291 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.611275    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241" path="/var/lib/kubelet/pods/e692d2fb-a7c5-4b9a-9e19-883eb44d4241/volumes"
Oct 30 00:30:20.677716 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.677678    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"]
Oct 30 00:30:20.678427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678404    4441 cpu_manager.go:414] "containerMap: removing container" podUID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" containerName="cleanup"
Oct 30 00:30:20.678427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678426    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" containerName="cleanup"
Oct 30 00:30:20.678608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678455    4441 cpu_manager.go:414] "containerMap: removing container" podUID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" containerName="cleanup"
Oct 30 00:30:20.678608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678461    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" containerName="cleanup"
Oct 30 00:30:20.678608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678479    4441 cpu_manager.go:414] "containerMap: removing container" podUID="7e0b85a3-27cc-4e62-917e-b524279557f5" containerName="cleanup"
Oct 30 00:30:20.678608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678486    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0b85a3-27cc-4e62-917e-b524279557f5" containerName="cleanup"
Oct 30 00:30:20.678608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678499    4441 cpu_manager.go:414] "containerMap: removing container" podUID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241" containerName="cleanup"
Oct 30 00:30:20.678608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678505    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241" containerName="cleanup"
Oct 30 00:30:20.678608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678519    4441 cpu_manager.go:414] "containerMap: removing container" podUID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241" containerName="cleanup"
Oct 30 00:30:20.678608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678526    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241" containerName="cleanup"
Oct 30 00:30:20.678608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678579    4441 cpu_manager.go:414] "containerMap: removing container" podUID="7e0b85a3-27cc-4e62-917e-b524279557f5" containerName="cleanup"
Oct 30 00:30:20.678608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678587    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0b85a3-27cc-4e62-917e-b524279557f5" containerName="cleanup"
Oct 30 00:30:20.679141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678732    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241" containerName="cleanup"
Oct 30 00:30:20.679141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678748    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" containerName="cleanup"
Oct 30 00:30:20.679141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678763    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0b85a3-27cc-4e62-917e-b524279557f5" containerName="cleanup"
Oct 30 00:30:20.679141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678774    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0b85a3-27cc-4e62-917e-b524279557f5" containerName="cleanup"
Oct 30 00:30:20.679141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678784    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" containerName="cleanup"
Oct 30 00:30:20.679141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678796    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" containerName="cleanup"
Oct 30 00:30:20.679141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.678806    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241" containerName="cleanup"
Oct 30 00:30:20.683774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.683751    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"
Oct 30 00:30:20.699618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.699592    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"]
Oct 30 00:30:20.705336 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.705299    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:30:20.748268 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.743465    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"
Oct 30 00:30:20.748268 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.743480    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884" event={"ID":"6c2563bc-4202-4d5c-92ea-15aaf4cd88ec","Type":"ContainerDied","Data":"7d092bbf082da720503667d03fcf68cf33e9a43528e2724dbaf7490e6839271f"}
Oct 30 00:30:20.748268 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.743554    4441 scope.go:122] "RemoveContainer" containerID="6b7fdadb4cc5ce6a0f94b7de193041154700bd9375f440545076aefd6943deb4"
Oct 30 00:30:20.749962 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.749931    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f64499a1-jlr8r\" (UID: \"b2557020-c735-4e8e-9d2a-09bc294079e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"
Oct 30 00:30:20.750125 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.750073    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f64499a1-jlr8r\" (UID: \"b2557020-c735-4e8e-9d2a-09bc294079e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"
Oct 30 00:30:20.750220 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.750132    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhpxl\" (UniqueName: \"kubernetes.io/projected/b2557020-c735-4e8e-9d2a-09bc294079e6-kube-api-access-fhpxl\") pod \"maas-api-revoke-sub-f64499a1-jlr8r\" (UID: \"b2557020-c735-4e8e-9d2a-09bc294079e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"
Oct 30 00:30:20.799096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.799060    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"]
Oct 30 00:30:20.808646 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.808612    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93016b-29888670-l4884"]
Oct 30 00:30:20.852901 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.852863    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f64499a1-jlr8r\" (UID: \"b2557020-c735-4e8e-9d2a-09bc294079e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"
Oct 30 00:30:20.853328 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.853288    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f64499a1-jlr8r\" (UID: \"b2557020-c735-4e8e-9d2a-09bc294079e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"
Oct 30 00:30:20.853569 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.853553    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhpxl\" (UniqueName: \"kubernetes.io/projected/b2557020-c735-4e8e-9d2a-09bc294079e6-kube-api-access-fhpxl\") pod \"maas-api-revoke-sub-f64499a1-jlr8r\" (UID: \"b2557020-c735-4e8e-9d2a-09bc294079e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"
Oct 30 00:30:20.855552 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.855514    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f64499a1-jlr8r\" (UID: \"b2557020-c735-4e8e-9d2a-09bc294079e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"
Oct 30 00:30:20.881920 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.881078    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhpxl\" (UniqueName: \"kubernetes.io/projected/b2557020-c735-4e8e-9d2a-09bc294079e6-kube-api-access-fhpxl\") pod \"maas-api-revoke-sub-f64499a1-jlr8r\" (UID: \"b2557020-c735-4e8e-9d2a-09bc294079e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"
Oct 30 00:30:20.881920 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.881840    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f64499a1-jlr8r\" (UID: \"b2557020-c735-4e8e-9d2a-09bc294079e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"
Oct 30 00:30:20.968662 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.968631    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"
Oct 30 00:30:20.997581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:20.997553    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"
Oct 30 00:30:21.057947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:21.056209    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83cfdc37-5f4d-4bcc-a667-da57a16f8301-kube-api-access-gg5xj\" (UniqueName: \"kubernetes.io/projected/83cfdc37-5f4d-4bcc-a667-da57a16f8301-kube-api-access-gg5xj\") pod \"83cfdc37-5f4d-4bcc-a667-da57a16f8301\" (UID: \"83cfdc37-5f4d-4bcc-a667-da57a16f8301\") "
Oct 30 00:30:21.057947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:21.056695    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83cfdc37-5f4d-4bcc-a667-da57a16f8301-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cfdc37-5f4d-4bcc-a667-da57a16f8301-maas-api-service-ca\") pod \"83cfdc37-5f4d-4bcc-a667-da57a16f8301\" (UID: \"83cfdc37-5f4d-4bcc-a667-da57a16f8301\") "
Oct 30 00:30:21.059184 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:21.058408    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83cfdc37-5f4d-4bcc-a667-da57a16f8301-maas-api-service-ca" pod "83cfdc37-5f4d-4bcc-a667-da57a16f8301" (UID: "83cfdc37-5f4d-4bcc-a667-da57a16f8301"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:30:21.060466 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:21.060436    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83cfdc37-5f4d-4bcc-a667-da57a16f8301-kube-api-access-gg5xj" pod "83cfdc37-5f4d-4bcc-a667-da57a16f8301" (UID: "83cfdc37-5f4d-4bcc-a667-da57a16f8301"). InnerVolumeSpecName "kube-api-access-gg5xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:30:21.159549 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:21.159412    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cfdc37-5f4d-4bcc-a667-da57a16f8301-maas-api-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:21.159549 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:21.159452    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5xj\" (UniqueName: \"kubernetes.io/projected/83cfdc37-5f4d-4bcc-a667-da57a16f8301-kube-api-access-gg5xj\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:21.339229 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:21.339202    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"]
Oct 30 00:30:21.344369 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:30:21.344333    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2557020_c735_4e8e_9d2a_09bc294079e6.slice/crio-eafecb22f437aeda242eb28f8ab4e85e609ceee40255cbd46446195375067081 WatchSource:0}: Error finding container eafecb22f437aeda242eb28f8ab4e85e609ceee40255cbd46446195375067081: Status 404 returned error can't find the container with id eafecb22f437aeda242eb28f8ab4e85e609ceee40255cbd46446195375067081
Oct 30 00:30:21.785390 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:21.785352    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r" event={"ID":"b2557020-c735-4e8e-9d2a-09bc294079e6","Type":"ContainerStarted","Data":"eafecb22f437aeda242eb28f8ab4e85e609ceee40255cbd46446195375067081"}
Oct 30 00:30:21.787911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:21.787873    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"
Oct 30 00:30:21.787911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:21.787902    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw" event={"ID":"83cfdc37-5f4d-4bcc-a667-da57a16f8301","Type":"ContainerDied","Data":"e6797d0078e760e7b31324fcd1a3517bddfe50cc62926acad7747553ae2b5706"}
Oct 30 00:30:21.788123 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:21.787961    4441 scope.go:122] "RemoveContainer" containerID="ee6ef11289ee8afe84281fa6ea5164225be1dbcb1444a0e9adf41abab6ed7e11"
Oct 30 00:30:21.832774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:21.832741    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"]
Oct 30 00:30:21.842812 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:21.842784    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-322d20-29888670-4phjw"]
Oct 30 00:30:22.613371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:22.613341    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" path="/var/lib/kubelet/pods/6c2563bc-4202-4d5c-92ea-15aaf4cd88ec/volumes"
Oct 30 00:30:22.613994 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:22.613969    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83cfdc37-5f4d-4bcc-a667-da57a16f8301" path="/var/lib/kubelet/pods/83cfdc37-5f4d-4bcc-a667-da57a16f8301/volumes"
Oct 30 00:30:22.797717 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:22.797682    4441 generic.go:356] "Generic (PLEG): container finished" podID="b2557020-c735-4e8e-9d2a-09bc294079e6" containerID="2150ebfce2f65f92c79723a3dda33ca8f2cc93c442043429f2a6aa7f27756834" exitCode=0
Oct 30 00:30:22.798416 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:22.797896    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r" event={"ID":"b2557020-c735-4e8e-9d2a-09bc294079e6","Type":"ContainerDied","Data":"2150ebfce2f65f92c79723a3dda33ca8f2cc93c442043429f2a6aa7f27756834"}
Oct 30 00:30:24.397050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:24.396101    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:30:24.771057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:24.755205    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:30:24.894053 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:24.860740    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:30:25.250248 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.245535    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:30:25.292048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.280664    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 00:30:25.337847 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.337795    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:30:25.411103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.409499    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:30:25.411103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.409443    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:30:25.427420 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.427385    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:30:25.468848 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.468823    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"
Oct 30 00:30:25.556622 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.556591    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2557020-c735-4e8e-9d2a-09bc294079e6-kube-api-access-fhpxl\" (UniqueName: \"kubernetes.io/projected/b2557020-c735-4e8e-9d2a-09bc294079e6-kube-api-access-fhpxl\") pod \"b2557020-c735-4e8e-9d2a-09bc294079e6\" (UID: \"b2557020-c735-4e8e-9d2a-09bc294079e6\") "
Oct 30 00:30:25.561246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.561214    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-xks-ca\") pod \"b2557020-c735-4e8e-9d2a-09bc294079e6\" (UID: \"b2557020-c735-4e8e-9d2a-09bc294079e6\") "
Oct 30 00:30:25.561575 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.561538    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-openshift-ca\") pod \"b2557020-c735-4e8e-9d2a-09bc294079e6\" (UID: \"b2557020-c735-4e8e-9d2a-09bc294079e6\") "
Oct 30 00:30:25.571615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.562162    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-openshift-ca" pod "b2557020-c735-4e8e-9d2a-09bc294079e6" (UID: "b2557020-c735-4e8e-9d2a-09bc294079e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:30:25.651787 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.651759    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:30:25.654568 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.654541    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-xks-ca" pod "b2557020-c735-4e8e-9d2a-09bc294079e6" (UID: "b2557020-c735-4e8e-9d2a-09bc294079e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:30:25.655380 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.655347    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2557020-c735-4e8e-9d2a-09bc294079e6-kube-api-access-fhpxl" pod "b2557020-c735-4e8e-9d2a-09bc294079e6" (UID: "b2557020-c735-4e8e-9d2a-09bc294079e6"). InnerVolumeSpecName "kube-api-access-fhpxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:30:25.674490 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.674408    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhpxl\" (UniqueName: \"kubernetes.io/projected/b2557020-c735-4e8e-9d2a-09bc294079e6-kube-api-access-fhpxl\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:25.674696 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.674677    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-xks-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:25.682859 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:25.681066    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2557020-c735-4e8e-9d2a-09bc294079e6-maas-api-openshift-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:26.388136 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:26.388099    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"
Oct 30 00:30:26.388391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:26.388357    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r" event={"ID":"b2557020-c735-4e8e-9d2a-09bc294079e6","Type":"ContainerDied","Data":"eafecb22f437aeda242eb28f8ab4e85e609ceee40255cbd46446195375067081"}
Oct 30 00:30:26.388469 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:26.388439    4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eafecb22f437aeda242eb28f8ab4e85e609ceee40255cbd46446195375067081"
Oct 30 00:30:28.710363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:28.710331    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"
Oct 30 00:30:32.890660 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.890618    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"]
Oct 30 00:30:32.891694 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891669    4441 cpu_manager.go:414] "containerMap: removing container" podUID="83cfdc37-5f4d-4bcc-a667-da57a16f8301" containerName="cleanup"
Oct 30 00:30:32.891694 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891690    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83cfdc37-5f4d-4bcc-a667-da57a16f8301" containerName="cleanup"
Oct 30 00:30:32.891837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891715    4441 cpu_manager.go:414] "containerMap: removing container" podUID="b2557020-c735-4e8e-9d2a-09bc294079e6" containerName="revoke-keys"
Oct 30 00:30:32.891837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891720    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2557020-c735-4e8e-9d2a-09bc294079e6" containerName="revoke-keys"
Oct 30 00:30:32.891837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891758    4441 cpu_manager.go:414] "containerMap: removing container" podUID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" containerName="cleanup"
Oct 30 00:30:32.891837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891763    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2563bc-4202-4d5c-92ea-15aaf4cd88ec" containerName="cleanup"
Oct 30 00:30:32.891837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891773    4441 cpu_manager.go:414] "containerMap: removing container" podUID="83cfdc37-5f4d-4bcc-a667-da57a16f8301" containerName="cleanup"
Oct 30 00:30:32.891837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891778    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83cfdc37-5f4d-4bcc-a667-da57a16f8301" containerName="cleanup"
Oct 30 00:30:32.891837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891785    4441 cpu_manager.go:414] "containerMap: removing container" podUID="7e0b85a3-27cc-4e62-917e-b524279557f5" containerName="cleanup"
Oct 30 00:30:32.891837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891789    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0b85a3-27cc-4e62-917e-b524279557f5" containerName="cleanup"
Oct 30 00:30:32.891837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891806    4441 cpu_manager.go:414] "containerMap: removing container" podUID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241" containerName="cleanup"
Oct 30 00:30:32.891837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891810    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241" containerName="cleanup"
Oct 30 00:30:32.892173 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891909    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="83cfdc37-5f4d-4bcc-a667-da57a16f8301" containerName="cleanup"
Oct 30 00:30:32.892173 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891918    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e692d2fb-a7c5-4b9a-9e19-883eb44d4241" containerName="cleanup"
Oct 30 00:30:32.892173 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891926    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="83cfdc37-5f4d-4bcc-a667-da57a16f8301" containerName="cleanup"
Oct 30 00:30:32.892173 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891937    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0b85a3-27cc-4e62-917e-b524279557f5" containerName="cleanup"
Oct 30 00:30:32.892173 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.891943    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2557020-c735-4e8e-9d2a-09bc294079e6" containerName="revoke-keys"
Oct 30 00:30:32.897322 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.897303    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:32.914440 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.914124    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"]
Oct 30 00:30:32.917556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.917361    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-1dd8c1-serving-cert\""
Oct 30 00:30:32.919165 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.919138    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbmsh\" (UniqueName: \"kubernetes.io/projected/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-kube-api-access-gbmsh\") pod \"maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:32.919424 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.919402    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:32.919735 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.919714    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-metrics-certs\") pod \"maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:32.919853 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:32.919838    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:33.021424 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:33.021381    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:33.021604 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:33.021481    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbmsh\" (UniqueName: \"kubernetes.io/projected/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-kube-api-access-gbmsh\") pod \"maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:33.021604 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:33.021531    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:33.021756 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:33.021678    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-metrics-certs\") pod \"maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:33.021756 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:33.021716    4441 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-serving-cert: secret "maas-api-e2e-praxis-a-1dd8c1-serving-cert" not found
Oct 30 00:30:33.021874 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:33.021783    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-maas-api-tls podName:8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24 nodeName:}" failed. No retries permitted until 2026-10-30 00:30:33.521763439 +0000 UTC m=+1543.047057302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-maas-api-tls") pod "maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz" (UID: "8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24") : secret "maas-api-e2e-praxis-a-1dd8c1-serving-cert" not found
Oct 30 00:30:33.022201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:33.022175    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:33.024283 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:33.024259    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-metrics-certs\") pod \"maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:33.061081 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:33.042873    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbmsh\" (UniqueName: \"kubernetes.io/projected/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-kube-api-access-gbmsh\") pod \"maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:33.530000 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:33.529946    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:33.532532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:33.532504    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:33.829379 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:33.829272    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:33.939133 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:30:33.936641    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:30:34.167536 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.167451    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"]
Oct 30 00:30:34.179124 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:30:34.179095    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec9b2fa_26ca_4cd6_877b_e5a94fa9ce24.slice/crio-4527b3307764a4d89566855fe3efad638e3c0ef90ef4264f2611e5ec33a3b63d WatchSource:0}: Error finding container 4527b3307764a4d89566855fe3efad638e3c0ef90ef4264f2611e5ec33a3b63d: Status 404 returned error can't find the container with id 4527b3307764a4d89566855fe3efad638e3c0ef90ef4264f2611e5ec33a3b63d
Oct 30 00:30:34.450178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.450073    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz" event={"ID":"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24","Type":"ContainerStarted","Data":"4527b3307764a4d89566855fe3efad638e3c0ef90ef4264f2611e5ec33a3b63d"}
Oct 30 00:30:34.575211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.575173    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"]
Oct 30 00:30:34.577929 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.576169    4441 cpu_manager.go:414] "containerMap: removing container" podUID="83cfdc37-5f4d-4bcc-a667-da57a16f8301" containerName="cleanup"
Oct 30 00:30:34.577929 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.576195    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83cfdc37-5f4d-4bcc-a667-da57a16f8301" containerName="cleanup"
Oct 30 00:30:34.577929 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.576456    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="83cfdc37-5f4d-4bcc-a667-da57a16f8301" containerName="cleanup"
Oct 30 00:30:34.582421 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.582399    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.593643 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.593615    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"]
Oct 30 00:30:34.598700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.598678    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-93016b-openshift-default-dockercfg-6ddrc\""
Oct 30 00:30:34.667440 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.667397    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15bdf08-77c7-4133-af8f-5dd59c119513-istiod-ca-cert\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.667642 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.667621    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-podinfo\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.667777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.667761    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kbkb\" (UniqueName: \"kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-kube-api-access-8kbkb\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.667984 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.667971    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-credential-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.668135 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.668118    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-data\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.668316 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.668301    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.668585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.668531    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-token\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.670140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.670112    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-certs\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.670290 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.670273    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-envoy\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.771890 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.771850    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15bdf08-77c7-4133-af8f-5dd59c119513-istiod-ca-cert\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.772174 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.771913    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-podinfo\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.772174 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.771955    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kbkb\" (UniqueName: \"kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-kube-api-access-8kbkb\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.772174 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.772138    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-credential-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.772174 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.772168    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-data\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.772412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.772240    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.772412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.772301    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-token\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.772412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.772342    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-certs\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.772412 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.772364    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-envoy\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.772656 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.772629    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-data\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.772710 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.772659    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15bdf08-77c7-4133-af8f-5dd59c119513-istiod-ca-cert\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.772710 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.772665    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-credential-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.772945 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.772921    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-certs\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.773018 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.772940    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.775109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.775083    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-envoy\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.775250 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.775091    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-podinfo\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.792516 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.792493    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-token\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.792625 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.792548    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kbkb\" (UniqueName: \"kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-kube-api-access-8kbkb\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:34.901615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:34.901567    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:35.235304 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:35.235273    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"]
Oct 30 00:30:35.236209 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:30:35.236183    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15bdf08_77c7_4133_af8f_5dd59c119513.slice/crio-52f8183471f7f731bca737934d82a6ff13f7f2b9ff1911badfcc4a1b8ccf08fb WatchSource:0}: Error finding container 52f8183471f7f731bca737934d82a6ff13f7f2b9ff1911badfcc4a1b8ccf08fb: Status 404 returned error can't find the container with id 52f8183471f7f731bca737934d82a6ff13f7f2b9ff1911badfcc4a1b8ccf08fb
Oct 30 00:30:35.239939 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:35.239900    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:30:35.240065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:35.239987    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:30:35.240065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:35.240048    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:30:35.462085 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:35.461975    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz" event={"ID":"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24","Type":"ContainerStarted","Data":"c65ae642e9535e3d75454cf457e69f3fbe0e1480b69d5b734e221537d2c89f91"}
Oct 30 00:30:35.462293 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:35.462101    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:35.464132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:35.464104    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz" event={"ID":"f15bdf08-77c7-4133-af8f-5dd59c119513","Type":"ContainerStarted","Data":"85b0af4ec03a0847e62a69206ba9d23588749f7e037440d9e3b5af3cefbef833"}
Oct 30 00:30:35.464265 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:35.464146    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz" event={"ID":"f15bdf08-77c7-4133-af8f-5dd59c119513","Type":"ContainerStarted","Data":"52f8183471f7f731bca737934d82a6ff13f7f2b9ff1911badfcc4a1b8ccf08fb"}
Oct 30 00:30:35.503335 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:35.503233    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz" podStartSLOduration=3.036355409 podStartE2EDuration="3.503217655s" totalImagesPullingTime="466.862246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:35.499275541 +0000 UTC m=+1545.024569419" watchObservedRunningTime="2026-10-30 00:30:35.503217655 +0000 UTC m=+1545.028511555"
Oct 30 00:30:35.540868 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:35.540814    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz" podStartSLOduration=1.540799376 podStartE2EDuration="1.540799376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:35.539838973 +0000 UTC m=+1545.065132852" watchObservedRunningTime="2026-10-30 00:30:35.540799376 +0000 UTC m=+1545.066093253"
Oct 30 00:30:35.902399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:35.902310    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:36.903799 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:36.903764    4441 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 00:30:36.904205 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:36.903815    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz" podUID="f15bdf08-77c7-4133-af8f-5dd59c119513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 00:30:37.902339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:37.902302    4441 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 00:30:37.902534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:37.902360    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz" podUID="f15bdf08-77c7-4133-af8f-5dd59c119513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 00:30:38.902557 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:38.902521    4441 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 00:30:38.902951 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:38.902581    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz" podUID="f15bdf08-77c7-4133-af8f-5dd59c119513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 00:30:39.159345 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:39.159253    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"]
Oct 30 00:30:39.181124 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:39.181089    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"]
Oct 30 00:30:39.905050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:39.904640    4441 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 00:30:39.905050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:39.904703    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz" podUID="f15bdf08-77c7-4133-af8f-5dd59c119513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 00:30:40.902426 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:40.902391    4441 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 00:30:40.902596 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:40.902445    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz" podUID="f15bdf08-77c7-4133-af8f-5dd59c119513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 00:30:41.477135 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:41.477105    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:30:41.901956 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:41.901861    4441 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 00:30:41.901956 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:41.901928    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz" podUID="f15bdf08-77c7-4133-af8f-5dd59c119513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 00:30:42.906569 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:42.906539    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:42.907014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:42.907001    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:42.907733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:42.907709    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:30:43.192480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.192393    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"]
Oct 30 00:30:43.192667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.192639    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb" podUID="e2f95d5b-fbe6-4ee5-be65-54f16728355c" containerName="maas-api" containerID="cri-o://078ce15de85c055ab6e1e889fd619a056918efb272038f3ee0f70e6335d9320c" gracePeriod=30
Oct 30 00:30:43.328841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.328806    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"]
Oct 30 00:30:43.329096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.329069    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw" podUID="e8480250-bbbd-44f3-a690-b5d2cb97efda" containerName="payload-processing" containerID="cri-o://fcb0962e3c8daa737e8405b3cca6729f57b63aaede9280bcdaba3f3cf46aa5bb" gracePeriod=30
Oct 30 00:30:43.526552 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.526516    4441 generic.go:356] "Generic (PLEG): container finished" podID="e8480250-bbbd-44f3-a690-b5d2cb97efda" containerID="fcb0962e3c8daa737e8405b3cca6729f57b63aaede9280bcdaba3f3cf46aa5bb" exitCode=0
Oct 30 00:30:43.526682 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.526667    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw" event={"ID":"e8480250-bbbd-44f3-a690-b5d2cb97efda","Type":"ContainerDied","Data":"fcb0962e3c8daa737e8405b3cca6729f57b63aaede9280bcdaba3f3cf46aa5bb"}
Oct 30 00:30:43.529569 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.528577    4441 generic.go:356] "Generic (PLEG): container finished" podID="e2f95d5b-fbe6-4ee5-be65-54f16728355c" containerID="078ce15de85c055ab6e1e889fd619a056918efb272038f3ee0f70e6335d9320c" exitCode=0
Oct 30 00:30:43.529569 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.529538    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb" event={"ID":"e2f95d5b-fbe6-4ee5-be65-54f16728355c","Type":"ContainerDied","Data":"078ce15de85c055ab6e1e889fd619a056918efb272038f3ee0f70e6335d9320c"}
Oct 30 00:30:43.827310 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.827267    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"
Oct 30 00:30:43.867714 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.867685    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.926895    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8480250-bbbd-44f3-a690-b5d2cb97efda-kube-api-access-cqfcj\" (UniqueName: \"kubernetes.io/projected/e8480250-bbbd-44f3-a690-b5d2cb97efda-kube-api-access-cqfcj\") pod \"e8480250-bbbd-44f3-a690-b5d2cb97efda\" (UID: \"e8480250-bbbd-44f3-a690-b5d2cb97efda\") "
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.926957    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-metrics-certs\") pod \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") "
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.927160    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8480250-bbbd-44f3-a690-b5d2cb97efda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8480250-bbbd-44f3-a690-b5d2cb97efda-plugins-config-volume\") pod \"e8480250-bbbd-44f3-a690-b5d2cb97efda\" (UID: \"e8480250-bbbd-44f3-a690-b5d2cb97efda\") "
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.927249    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-maas-api-tls\") pod \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") "
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.927338    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f95d5b-fbe6-4ee5-be65-54f16728355c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f95d5b-fbe6-4ee5-be65-54f16728355c-openshift-service-ca\") pod \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") "
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.927414    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f95d5b-fbe6-4ee5-be65-54f16728355c-kube-api-access-ks96f\" (UniqueName: \"kubernetes.io/projected/e2f95d5b-fbe6-4ee5-be65-54f16728355c-kube-api-access-ks96f\") pod \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\" (UID: \"e2f95d5b-fbe6-4ee5-be65-54f16728355c\") "
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.928004    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f95d5b-fbe6-4ee5-be65-54f16728355c-openshift-service-ca" pod "e2f95d5b-fbe6-4ee5-be65-54f16728355c" (UID: "e2f95d5b-fbe6-4ee5-be65-54f16728355c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.928008    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8480250-bbbd-44f3-a690-b5d2cb97efda-plugins-config-volume" pod "e8480250-bbbd-44f3-a690-b5d2cb97efda" (UID: "e8480250-bbbd-44f3-a690-b5d2cb97efda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.928772    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8480250-bbbd-44f3-a690-b5d2cb97efda-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.928797    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f95d5b-fbe6-4ee5-be65-54f16728355c-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.930775    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f95d5b-fbe6-4ee5-be65-54f16728355c-kube-api-access-ks96f" pod "e2f95d5b-fbe6-4ee5-be65-54f16728355c" (UID: "e2f95d5b-fbe6-4ee5-be65-54f16728355c"). InnerVolumeSpecName "kube-api-access-ks96f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.931297    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-metrics-certs" pod "e2f95d5b-fbe6-4ee5-be65-54f16728355c" (UID: "e2f95d5b-fbe6-4ee5-be65-54f16728355c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.931585    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8480250-bbbd-44f3-a690-b5d2cb97efda-kube-api-access-cqfcj" pod "e8480250-bbbd-44f3-a690-b5d2cb97efda" (UID: "e8480250-bbbd-44f3-a690-b5d2cb97efda"). InnerVolumeSpecName "kube-api-access-cqfcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:30:43.941090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:43.933477    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-maas-api-tls" pod "e2f95d5b-fbe6-4ee5-be65-54f16728355c" (UID: "e2f95d5b-fbe6-4ee5-be65-54f16728355c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:30:44.030855 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.030756    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:44.030855 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.030797    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks96f\" (UniqueName: \"kubernetes.io/projected/e2f95d5b-fbe6-4ee5-be65-54f16728355c-kube-api-access-ks96f\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:44.030855 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.030807    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqfcj\" (UniqueName: \"kubernetes.io/projected/e8480250-bbbd-44f3-a690-b5d2cb97efda-kube-api-access-cqfcj\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:44.030855 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.030819    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f95d5b-fbe6-4ee5-be65-54f16728355c-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:44.164304 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.162563    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8" podUID="55345590-08a0-4f55-b6f5-4768c149ad8b" containerName="payload-processing" containerID="cri-o://2a103d7f137d0c469e0fe658b647888d27f0514e0dac62ac39767bb349fe67eb" gracePeriod=25
Oct 30 00:30:44.181667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.181625    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj" podUID="a9e91820-48d8-494c-a1b4-d18db107bfe8" containerName="payload-pre-processing" containerID="cri-o://274f0217cb55a4c4faf9e027489ee4e5a32ffcea0c447dea6eff4cc6e1bd3c38" gracePeriod=25
Oct 30 00:30:44.541262 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.541230    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"
Oct 30 00:30:44.541428 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.541224    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb" event={"ID":"e2f95d5b-fbe6-4ee5-be65-54f16728355c","Type":"ContainerDied","Data":"3b22ff87c2d278b5bdb8369102878fd18696029fd88228739976f25eb3bcaeb0"}
Oct 30 00:30:44.541428 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.541421    4441 scope.go:122] "RemoveContainer" containerID="078ce15de85c055ab6e1e889fd619a056918efb272038f3ee0f70e6335d9320c"
Oct 30 00:30:44.544099 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.544057    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw" event={"ID":"e8480250-bbbd-44f3-a690-b5d2cb97efda","Type":"ContainerDied","Data":"48b8500ee75318a8c82b407572f867a2f792f84746f963adc8902c692043a46c"}
Oct 30 00:30:44.544099 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.544085    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"
Oct 30 00:30:44.554931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.554911    4441 scope.go:122] "RemoveContainer" containerID="fcb0962e3c8daa737e8405b3cca6729f57b63aaede9280bcdaba3f3cf46aa5bb"
Oct 30 00:30:44.612118 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.612015    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"]
Oct 30 00:30:44.620697 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.620671    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-7b5fc7688-gvqjw"]
Oct 30 00:30:44.644974 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.644930    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"]
Oct 30 00:30:44.650600 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:44.650565    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb3ad-7cd7bd7c96-vx2hb"]
Oct 30 00:30:45.530286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:45.530245    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:30:46.611882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:46.611829    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f95d5b-fbe6-4ee5-be65-54f16728355c" path="/var/lib/kubelet/pods/e2f95d5b-fbe6-4ee5-be65-54f16728355c/volumes"
Oct 30 00:30:46.612587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:46.612509    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8480250-bbbd-44f3-a690-b5d2cb97efda" path="/var/lib/kubelet/pods/e8480250-bbbd-44f3-a690-b5d2cb97efda/volumes"
Oct 30 00:30:47.374187 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:47.374155    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8" podUID="55345590-08a0-4f55-b6f5-4768c149ad8b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:30:48.400089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:48.400053    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj" podUID="a9e91820-48d8-494c-a1b4-d18db107bfe8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:30:57.373910 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:57.373877    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8" podUID="55345590-08a0-4f55-b6f5-4768c149ad8b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:30:58.399179 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:58.399136    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj" podUID="a9e91820-48d8-494c-a1b4-d18db107bfe8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:30:59.144768 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.144733    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"]
Oct 30 00:30:59.145044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.144975    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m" podUID="bfb1eed7-aaad-47b4-9bf7-005a5403f0af" containerName="maas-api" containerID="cri-o://46cfc9cb1947101b34936358cc57f92015e6560d69246247dd9f32de9fa14e3d" gracePeriod=30
Oct 30 00:30:59.667987 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.667956    4441 generic.go:356] "Generic (PLEG): container finished" podID="bfb1eed7-aaad-47b4-9bf7-005a5403f0af" containerID="46cfc9cb1947101b34936358cc57f92015e6560d69246247dd9f32de9fa14e3d" exitCode=0
Oct 30 00:30:59.668381 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.668063    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m" event={"ID":"bfb1eed7-aaad-47b4-9bf7-005a5403f0af","Type":"ContainerDied","Data":"46cfc9cb1947101b34936358cc57f92015e6560d69246247dd9f32de9fa14e3d"}
Oct 30 00:30:59.668381 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.668125    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m" event={"ID":"bfb1eed7-aaad-47b4-9bf7-005a5403f0af","Type":"ContainerDied","Data":"01b3540961741a908fe17a43365b84f90bcc7ad3d5374ce1f87bb77f7f050e7e"}
Oct 30 00:30:59.668381 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.668148    4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01b3540961741a908fe17a43365b84f90bcc7ad3d5374ce1f87bb77f7f050e7e"
Oct 30 00:30:59.677129 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.677067    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:30:59.695585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.695557    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-kube-api-access-rvf6z\" (UniqueName: \"kubernetes.io/projected/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-kube-api-access-rvf6z\") pod \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") "
Oct 30 00:30:59.695585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.695589    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-openshift-service-ca\") pod \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") "
Oct 30 00:30:59.695836 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.695815    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-metrics-certs\") pod \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") "
Oct 30 00:30:59.695910 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.695868    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-maas-api-tls\") pod \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\" (UID: \"bfb1eed7-aaad-47b4-9bf7-005a5403f0af\") "
Oct 30 00:30:59.696103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.696063    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-openshift-service-ca" pod "bfb1eed7-aaad-47b4-9bf7-005a5403f0af" (UID: "bfb1eed7-aaad-47b4-9bf7-005a5403f0af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:30:59.696580 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.696556    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:59.698457 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.698431    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-kube-api-access-rvf6z" pod "bfb1eed7-aaad-47b4-9bf7-005a5403f0af" (UID: "bfb1eed7-aaad-47b4-9bf7-005a5403f0af"). InnerVolumeSpecName "kube-api-access-rvf6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:30:59.698613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.698527    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-maas-api-tls" pod "bfb1eed7-aaad-47b4-9bf7-005a5403f0af" (UID: "bfb1eed7-aaad-47b4-9bf7-005a5403f0af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:30:59.698718 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.698620    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-metrics-certs" pod "bfb1eed7-aaad-47b4-9bf7-005a5403f0af" (UID: "bfb1eed7-aaad-47b4-9bf7-005a5403f0af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:30:59.799508 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.799374    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:59.799508 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.799420    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:30:59.799508 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:30:59.799439    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvf6z\" (UniqueName: \"kubernetes.io/projected/bfb1eed7-aaad-47b4-9bf7-005a5403f0af-kube-api-access-rvf6z\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:31:00.675324 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:00.675289    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"
Oct 30 00:31:00.722079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:00.722041    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"]
Oct 30 00:31:00.730950 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:00.730912    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93016b-5bb75bc976-7qh9m"]
Oct 30 00:31:02.320895 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.320863    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"]
Oct 30 00:31:02.321501 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.321485    4441 cpu_manager.go:414] "containerMap: removing container" podUID="e8480250-bbbd-44f3-a690-b5d2cb97efda" containerName="payload-processing"
Oct 30 00:31:02.321553 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.321503    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8480250-bbbd-44f3-a690-b5d2cb97efda" containerName="payload-processing"
Oct 30 00:31:02.321553 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.321528    4441 cpu_manager.go:414] "containerMap: removing container" podUID="e2f95d5b-fbe6-4ee5-be65-54f16728355c" containerName="maas-api"
Oct 30 00:31:02.321553 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.321533    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f95d5b-fbe6-4ee5-be65-54f16728355c" containerName="maas-api"
Oct 30 00:31:02.321553 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.321550    4441 cpu_manager.go:414] "containerMap: removing container" podUID="bfb1eed7-aaad-47b4-9bf7-005a5403f0af" containerName="maas-api"
Oct 30 00:31:02.321673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.321556    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb1eed7-aaad-47b4-9bf7-005a5403f0af" containerName="maas-api"
Oct 30 00:31:02.321673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.321671    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8480250-bbbd-44f3-a690-b5d2cb97efda" containerName="payload-processing"
Oct 30 00:31:02.321737 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.321685    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f95d5b-fbe6-4ee5-be65-54f16728355c" containerName="maas-api"
Oct 30 00:31:02.321737 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.321697    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb1eed7-aaad-47b4-9bf7-005a5403f0af" containerName="maas-api"
Oct 30 00:31:02.325208 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.325185    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"
Oct 30 00:31:02.346016 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.345986    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"]
Oct 30 00:31:02.349281 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.349178    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-bc8b7d\""
Oct 30 00:31:02.349281 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.349252    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-bc8b7d-dockercfg-jxqd7\""
Oct 30 00:31:02.447066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.446564    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8030ad10-9610-4159-a607-92ca15758e63-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5\" (UID: \"8030ad10-9610-4159-a607-92ca15758e63\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"
Oct 30 00:31:02.447066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.446612    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skl89\" (UniqueName: \"kubernetes.io/projected/8030ad10-9610-4159-a607-92ca15758e63-kube-api-access-skl89\") pod \"payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5\" (UID: \"8030ad10-9610-4159-a607-92ca15758e63\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"
Oct 30 00:31:02.550095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.527096    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"]
Oct 30 00:31:02.550095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.533381    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"
Oct 30 00:31:02.550095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.548236    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98b69041-6f23-4680-a25f-2d38b2228ed3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm\" (UID: \"98b69041-6f23-4680-a25f-2d38b2228ed3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"
Oct 30 00:31:02.550095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.548343    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8030ad10-9610-4159-a607-92ca15758e63-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5\" (UID: \"8030ad10-9610-4159-a607-92ca15758e63\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"
Oct 30 00:31:02.550095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.548378    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skl89\" (UniqueName: \"kubernetes.io/projected/8030ad10-9610-4159-a607-92ca15758e63-kube-api-access-skl89\") pod \"payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5\" (UID: \"8030ad10-9610-4159-a607-92ca15758e63\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"
Oct 30 00:31:02.550095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.548558    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cwhm\" (UniqueName: \"kubernetes.io/projected/98b69041-6f23-4680-a25f-2d38b2228ed3-kube-api-access-9cwhm\") pod \"payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm\" (UID: \"98b69041-6f23-4680-a25f-2d38b2228ed3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"
Oct 30 00:31:02.550095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.549542    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8030ad10-9610-4159-a607-92ca15758e63-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5\" (UID: \"8030ad10-9610-4159-a607-92ca15758e63\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"
Oct 30 00:31:02.551559 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.551486    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"]
Oct 30 00:31:02.588302 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.585052    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skl89\" (UniqueName: \"kubernetes.io/projected/8030ad10-9610-4159-a607-92ca15758e63-kube-api-access-skl89\") pod \"payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5\" (UID: \"8030ad10-9610-4159-a607-92ca15758e63\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"
Oct 30 00:31:02.636052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.636002    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb1eed7-aaad-47b4-9bf7-005a5403f0af" path="/var/lib/kubelet/pods/bfb1eed7-aaad-47b4-9bf7-005a5403f0af/volumes"
Oct 30 00:31:02.643556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.643531    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"
Oct 30 00:31:02.650689 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.650656    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cwhm\" (UniqueName: \"kubernetes.io/projected/98b69041-6f23-4680-a25f-2d38b2228ed3-kube-api-access-9cwhm\") pod \"payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm\" (UID: \"98b69041-6f23-4680-a25f-2d38b2228ed3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"
Oct 30 00:31:02.651312 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.651284    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98b69041-6f23-4680-a25f-2d38b2228ed3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm\" (UID: \"98b69041-6f23-4680-a25f-2d38b2228ed3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"
Oct 30 00:31:02.652530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.652508    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98b69041-6f23-4680-a25f-2d38b2228ed3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm\" (UID: \"98b69041-6f23-4680-a25f-2d38b2228ed3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"
Oct 30 00:31:02.677739 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.677714    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cwhm\" (UniqueName: \"kubernetes.io/projected/98b69041-6f23-4680-a25f-2d38b2228ed3-kube-api-access-9cwhm\") pod \"payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm\" (UID: \"98b69041-6f23-4680-a25f-2d38b2228ed3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"
Oct 30 00:31:02.884296 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.884205    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"
Oct 30 00:31:02.975974 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:02.975944    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"]
Oct 30 00:31:02.984957 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:31:02.984927    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8030ad10_9610_4159_a607_92ca15758e63.slice/crio-87ae576048388d8aa216b94366cce6c8a18ed59b299035df5f2ad2f960b36383 WatchSource:0}: Error finding container 87ae576048388d8aa216b94366cce6c8a18ed59b299035df5f2ad2f960b36383: Status 404 returned error can't find the container with id 87ae576048388d8aa216b94366cce6c8a18ed59b299035df5f2ad2f960b36383
Oct 30 00:31:03.189234 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:03.189206    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"]
Oct 30 00:31:03.190071 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:31:03.190003    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b69041_6f23_4680_a25f_2d38b2228ed3.slice/crio-2c48d265073167b63fa0278a9d3b226e94606dff86f0dc78d55d00c02f3ecb26 WatchSource:0}: Error finding container 2c48d265073167b63fa0278a9d3b226e94606dff86f0dc78d55d00c02f3ecb26: Status 404 returned error can't find the container with id 2c48d265073167b63fa0278a9d3b226e94606dff86f0dc78d55d00c02f3ecb26
Oct 30 00:31:03.703515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:03.703477    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5" event={"ID":"8030ad10-9610-4159-a607-92ca15758e63","Type":"ContainerStarted","Data":"c758c6dc5f1f08396c20a0a94d2f4a6067f9b036df7fab9f660162e98024993e"}
Oct 30 00:31:03.704007 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:03.703576    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5" event={"ID":"8030ad10-9610-4159-a607-92ca15758e63","Type":"ContainerStarted","Data":"87ae576048388d8aa216b94366cce6c8a18ed59b299035df5f2ad2f960b36383"}
Oct 30 00:31:03.704007 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:03.703807    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"
Oct 30 00:31:03.705336 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:03.705305    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm" event={"ID":"98b69041-6f23-4680-a25f-2d38b2228ed3","Type":"ContainerStarted","Data":"033aad24f18a74d3eb52e82df3e2eb633d534b978c0d59ef0fe82dcfb3afa38f"}
Oct 30 00:31:03.705458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:03.705359    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm" event={"ID":"98b69041-6f23-4680-a25f-2d38b2228ed3","Type":"ContainerStarted","Data":"2c48d265073167b63fa0278a9d3b226e94606dff86f0dc78d55d00c02f3ecb26"}
Oct 30 00:31:03.705458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:03.705403    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"
Oct 30 00:31:03.741264 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:03.741201    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5" podStartSLOduration=1.741181184 podStartE2EDuration="1.741181184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:03.738346574 +0000 UTC m=+1573.263640475" watchObservedRunningTime="2026-10-30 00:31:03.741181184 +0000 UTC m=+1573.266475060"
Oct 30 00:31:03.775733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:03.775674    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm" podStartSLOduration=1.775661154 podStartE2EDuration="1.775661154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:03.773105689 +0000 UTC m=+1573.298399567" watchObservedRunningTime="2026-10-30 00:31:03.775661154 +0000 UTC m=+1573.300955030"
Oct 30 00:31:03.921431 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:31:03.921317    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:31:04.481445 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.481421    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"
Oct 30 00:31:04.485495 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.485466    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55345590-08a0-4f55-b6f5-4768c149ad8b-kube-api-access-v4c26\" (UniqueName: \"kubernetes.io/projected/55345590-08a0-4f55-b6f5-4768c149ad8b-kube-api-access-v4c26\") pod \"55345590-08a0-4f55-b6f5-4768c149ad8b\" (UID: \"55345590-08a0-4f55-b6f5-4768c149ad8b\") "
Oct 30 00:31:04.485697 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.485680    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55345590-08a0-4f55-b6f5-4768c149ad8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55345590-08a0-4f55-b6f5-4768c149ad8b-plugins-config-volume\") pod \"55345590-08a0-4f55-b6f5-4768c149ad8b\" (UID: \"55345590-08a0-4f55-b6f5-4768c149ad8b\") "
Oct 30 00:31:04.486109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.486078    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55345590-08a0-4f55-b6f5-4768c149ad8b-plugins-config-volume" pod "55345590-08a0-4f55-b6f5-4768c149ad8b" (UID: "55345590-08a0-4f55-b6f5-4768c149ad8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:31:04.486470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.486449    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55345590-08a0-4f55-b6f5-4768c149ad8b-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:31:04.488002 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.487977    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55345590-08a0-4f55-b6f5-4768c149ad8b-kube-api-access-v4c26" pod "55345590-08a0-4f55-b6f5-4768c149ad8b" (UID: "55345590-08a0-4f55-b6f5-4768c149ad8b"). InnerVolumeSpecName "kube-api-access-v4c26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:31:04.498236 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.498213    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"
Oct 30 00:31:04.587200 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.587165    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e91820-48d8-494c-a1b4-d18db107bfe8-kube-api-access-bvkb4\" (UniqueName: \"kubernetes.io/projected/a9e91820-48d8-494c-a1b4-d18db107bfe8-kube-api-access-bvkb4\") pod \"a9e91820-48d8-494c-a1b4-d18db107bfe8\" (UID: \"a9e91820-48d8-494c-a1b4-d18db107bfe8\") "
Oct 30 00:31:04.587381 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.587224    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e91820-48d8-494c-a1b4-d18db107bfe8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e91820-48d8-494c-a1b4-d18db107bfe8-plugins-config-volume\") pod \"a9e91820-48d8-494c-a1b4-d18db107bfe8\" (UID: \"a9e91820-48d8-494c-a1b4-d18db107bfe8\") "
Oct 30 00:31:04.587703 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.587670    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e91820-48d8-494c-a1b4-d18db107bfe8-plugins-config-volume" pod "a9e91820-48d8-494c-a1b4-d18db107bfe8" (UID: "a9e91820-48d8-494c-a1b4-d18db107bfe8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:31:04.587988 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.587967    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4c26\" (UniqueName: \"kubernetes.io/projected/55345590-08a0-4f55-b6f5-4768c149ad8b-kube-api-access-v4c26\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:31:04.588113 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.587996    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e91820-48d8-494c-a1b4-d18db107bfe8-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:31:04.589515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.589488    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e91820-48d8-494c-a1b4-d18db107bfe8-kube-api-access-bvkb4" pod "a9e91820-48d8-494c-a1b4-d18db107bfe8" (UID: "a9e91820-48d8-494c-a1b4-d18db107bfe8"). InnerVolumeSpecName "kube-api-access-bvkb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:31:04.689372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.689341    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvkb4\" (UniqueName: \"kubernetes.io/projected/a9e91820-48d8-494c-a1b4-d18db107bfe8-kube-api-access-bvkb4\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:31:04.715380 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.715297    4441 generic.go:356] "Generic (PLEG): container finished" podID="a9e91820-48d8-494c-a1b4-d18db107bfe8" containerID="274f0217cb55a4c4faf9e027489ee4e5a32ffcea0c447dea6eff4cc6e1bd3c38" exitCode=0
Oct 30 00:31:04.715380 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.715357    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"
Oct 30 00:31:04.715917 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.715383    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj" event={"ID":"a9e91820-48d8-494c-a1b4-d18db107bfe8","Type":"ContainerDied","Data":"274f0217cb55a4c4faf9e027489ee4e5a32ffcea0c447dea6eff4cc6e1bd3c38"}
Oct 30 00:31:04.715917 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.715442    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj" event={"ID":"a9e91820-48d8-494c-a1b4-d18db107bfe8","Type":"ContainerDied","Data":"97c381dd8ba7d72f3cd86b565d0de673de65c74bb37adcdfb5625db1043f018f"}
Oct 30 00:31:04.715917 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.715474    4441 scope.go:122] "RemoveContainer" containerID="274f0217cb55a4c4faf9e027489ee4e5a32ffcea0c447dea6eff4cc6e1bd3c38"
Oct 30 00:31:04.717114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.717089    4441 generic.go:356] "Generic (PLEG): container finished" podID="55345590-08a0-4f55-b6f5-4768c149ad8b" containerID="2a103d7f137d0c469e0fe658b647888d27f0514e0dac62ac39767bb349fe67eb" exitCode=0
Oct 30 00:31:04.717235 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.717144    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"
Oct 30 00:31:04.718260 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.718235    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8" event={"ID":"55345590-08a0-4f55-b6f5-4768c149ad8b","Type":"ContainerDied","Data":"2a103d7f137d0c469e0fe658b647888d27f0514e0dac62ac39767bb349fe67eb"}
Oct 30 00:31:04.718385 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.718282    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8" event={"ID":"55345590-08a0-4f55-b6f5-4768c149ad8b","Type":"ContainerDied","Data":"361cbbac0b7f11560134ecb3c0d8a1ec5aff04e485bdcc34cfaaafbc2717f11f"}
Oct 30 00:31:04.726388 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.726369    4441 scope.go:122] "RemoveContainer" containerID="274f0217cb55a4c4faf9e027489ee4e5a32ffcea0c447dea6eff4cc6e1bd3c38"
Oct 30 00:31:04.726729 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:31:04.726706    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274f0217cb55a4c4faf9e027489ee4e5a32ffcea0c447dea6eff4cc6e1bd3c38\": container with ID starting with 274f0217cb55a4c4faf9e027489ee4e5a32ffcea0c447dea6eff4cc6e1bd3c38 not found: ID does not exist" containerID="274f0217cb55a4c4faf9e027489ee4e5a32ffcea0c447dea6eff4cc6e1bd3c38"
Oct 30 00:31:04.726788 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.726739    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274f0217cb55a4c4faf9e027489ee4e5a32ffcea0c447dea6eff4cc6e1bd3c38"} err="failed to get container status \"274f0217cb55a4c4faf9e027489ee4e5a32ffcea0c447dea6eff4cc6e1bd3c38\": rpc error: code = NotFound desc = could not find container \"274f0217cb55a4c4faf9e027489ee4e5a32ffcea0c447dea6eff4cc6e1bd3c38\": container with ID starting with 274f0217cb55a4c4faf9e027489ee4e5a32ffcea0c447dea6eff4cc6e1bd3c38 not found: ID does not exist"
Oct 30 00:31:04.726788 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.726757    4441 scope.go:122] "RemoveContainer" containerID="2a103d7f137d0c469e0fe658b647888d27f0514e0dac62ac39767bb349fe67eb"
Oct 30 00:31:04.737937 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.737915    4441 scope.go:122] "RemoveContainer" containerID="2a103d7f137d0c469e0fe658b647888d27f0514e0dac62ac39767bb349fe67eb"
Oct 30 00:31:04.738334 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:31:04.738314    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a103d7f137d0c469e0fe658b647888d27f0514e0dac62ac39767bb349fe67eb\": container with ID starting with 2a103d7f137d0c469e0fe658b647888d27f0514e0dac62ac39767bb349fe67eb not found: ID does not exist" containerID="2a103d7f137d0c469e0fe658b647888d27f0514e0dac62ac39767bb349fe67eb"
Oct 30 00:31:04.738408 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.738347    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a103d7f137d0c469e0fe658b647888d27f0514e0dac62ac39767bb349fe67eb"} err="failed to get container status \"2a103d7f137d0c469e0fe658b647888d27f0514e0dac62ac39767bb349fe67eb\": rpc error: code = NotFound desc = could not find container \"2a103d7f137d0c469e0fe658b647888d27f0514e0dac62ac39767bb349fe67eb\": container with ID starting with 2a103d7f137d0c469e0fe658b647888d27f0514e0dac62ac39767bb349fe67eb not found: ID does not exist"
Oct 30 00:31:04.763611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.763581    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"]
Oct 30 00:31:04.768151 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.768124    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93016b-6885b557ff-tm5rj"]
Oct 30 00:31:04.799271 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.799240    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"]
Oct 30 00:31:04.803201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:04.803172    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93016b-5469498f46-gp4j8"]
Oct 30 00:31:06.610015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:06.609982    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55345590-08a0-4f55-b6f5-4768c149ad8b" path="/var/lib/kubelet/pods/55345590-08a0-4f55-b6f5-4768c149ad8b/volumes"
Oct 30 00:31:06.610573 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:06.610555    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e91820-48d8-494c-a1b4-d18db107bfe8" path="/var/lib/kubelet/pods/a9e91820-48d8-494c-a1b4-d18db107bfe8/volumes"
Oct 30 00:31:09.405814 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.405763    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"]
Oct 30 00:31:09.406572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.406551    4441 cpu_manager.go:414] "containerMap: removing container" podUID="55345590-08a0-4f55-b6f5-4768c149ad8b" containerName="payload-processing"
Oct 30 00:31:09.406572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.406573    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55345590-08a0-4f55-b6f5-4768c149ad8b" containerName="payload-processing"
Oct 30 00:31:09.406710 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.406592    4441 cpu_manager.go:414] "containerMap: removing container" podUID="a9e91820-48d8-494c-a1b4-d18db107bfe8" containerName="payload-pre-processing"
Oct 30 00:31:09.406710 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.406598    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e91820-48d8-494c-a1b4-d18db107bfe8" containerName="payload-pre-processing"
Oct 30 00:31:09.406710 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.406698    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e91820-48d8-494c-a1b4-d18db107bfe8" containerName="payload-pre-processing"
Oct 30 00:31:09.406868 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.406715    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="55345590-08a0-4f55-b6f5-4768c149ad8b" containerName="payload-processing"
Oct 30 00:31:09.412009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.411989    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.425911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.425883    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"]
Oct 30 00:31:09.431811 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.431785    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.436602 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.436567    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"]
Oct 30 00:31:09.460962 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.460921    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"]
Oct 30 00:31:09.475984 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.475930    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.476178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.475999    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.476178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476060    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.476178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476099    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.476178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476133    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.476178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476166    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.476488 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476289    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.476488 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476333    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.476488 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476362    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a713f91-8d61-4d25-84d1-8121a511bd94-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.476488 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476403    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.476488 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476463    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.476761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476511    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.476761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476538    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.476761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476589    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.476761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476625    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.476761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476665    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.476761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476752    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8zdl\" (UniqueName: \"kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-kube-api-access-h8zdl\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.477203 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.476779    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpxlv\" (UniqueName: \"kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-kube-api-access-bpxlv\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.577867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.577833    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8zdl\" (UniqueName: \"kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-kube-api-access-h8zdl\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.577867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.577872    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpxlv\" (UniqueName: \"kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-kube-api-access-bpxlv\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.578132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.577944    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.578132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.577982    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.578132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578040    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.578132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578077    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.578132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578107    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.578402 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578138    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.578402 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578261    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.578402 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578309    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.578402 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578343    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a713f91-8d61-4d25-84d1-8121a511bd94-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.578402 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578378    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578480    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578483    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578575    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578607    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578663    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578707    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578746    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578753    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578756    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578858    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.578956    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.579123    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.579232    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.579353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.579298    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.580345 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.579420    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a713f91-8d61-4d25-84d1-8121a511bd94-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.580345 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.579438    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.580919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.580896    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.581071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.580964    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.581525 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.581497    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.581639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.581585    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.599405 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.599374    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.599792 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.599760    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.600071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.600016    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8zdl\" (UniqueName: \"kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-kube-api-access-h8zdl\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.600196 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.600171    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpxlv\" (UniqueName: \"kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-kube-api-access-bpxlv\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:09.727956 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.727921    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:09.758112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:09.758080    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:10.060627 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:10.060599    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"]
Oct 30 00:31:10.063231 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:31:10.063203    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a713f91_8d61_4d25_84d1_8121a511bd94.slice/crio-f3eb4e1bed681b2221aa61e9f44ddc9f61cbc4ca6658cdbdd8080bb56456f284 WatchSource:0}: Error finding container f3eb4e1bed681b2221aa61e9f44ddc9f61cbc4ca6658cdbdd8080bb56456f284: Status 404 returned error can't find the container with id f3eb4e1bed681b2221aa61e9f44ddc9f61cbc4ca6658cdbdd8080bb56456f284
Oct 30 00:31:10.066394 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:10.066358    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:31:10.066479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:10.066441    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:31:10.066520 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:10.066484    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:31:10.100329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:10.100302    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"]
Oct 30 00:31:10.101316 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:31:10.101293    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2cdb408_2ee3_44a9_8926_1bb6cc8e0edd.slice/crio-75fecb4775098526adcc20ccb8ee41c4654e08e1ac31933044b96abb151f4750 WatchSource:0}: Error finding container 75fecb4775098526adcc20ccb8ee41c4654e08e1ac31933044b96abb151f4750: Status 404 returned error can't find the container with id 75fecb4775098526adcc20ccb8ee41c4654e08e1ac31933044b96abb151f4750
Oct 30 00:31:10.104210 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:10.104176    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:31:10.104296 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:10.104248    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:31:10.104296 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:10.104277    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:31:10.785191 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:10.785144    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj" event={"ID":"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd","Type":"ContainerStarted","Data":"4526369ccd7d8c45da5281c742c3015ea2ff60105488ac69902eca9d29b5bc96"}
Oct 30 00:31:10.786220 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:10.785476    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj" event={"ID":"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd","Type":"ContainerStarted","Data":"75fecb4775098526adcc20ccb8ee41c4654e08e1ac31933044b96abb151f4750"}
Oct 30 00:31:10.787468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:10.787440    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth" event={"ID":"4a713f91-8d61-4d25-84d1-8121a511bd94","Type":"ContainerStarted","Data":"1c31a27cb70a0863a5477e9973186b0540b45fcb2369d9996d6a405970730fdc"}
Oct 30 00:31:10.787593 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:10.787491    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth" event={"ID":"4a713f91-8d61-4d25-84d1-8121a511bd94","Type":"ContainerStarted","Data":"f3eb4e1bed681b2221aa61e9f44ddc9f61cbc4ca6658cdbdd8080bb56456f284"}
Oct 30 00:31:10.871543 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:10.871475    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj" podStartSLOduration=1.871453853 podStartE2EDuration="1.871453853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:10.857593966 +0000 UTC m=+1580.382887838" watchObservedRunningTime="2026-10-30 00:31:10.871453853 +0000 UTC m=+1580.396747734"
Oct 30 00:31:11.729105 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:11.728208    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:11.734543 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:11.734152    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 00:31:11.734543 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:11.734218    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth" podUID="4a713f91-8d61-4d25-84d1-8121a511bd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 00:31:11.766267 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:11.766007    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:11.766267 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:11.766159    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 00:31:11.766267 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:11.766203    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj" podUID="c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 00:31:12.730568 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:12.730523    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 00:31:12.750678 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:12.750636    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth" podUID="4a713f91-8d61-4d25-84d1-8121a511bd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 00:31:12.773926 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:12.771165    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 00:31:12.773926 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:12.771259    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj" podUID="c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 00:31:13.740051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:13.730472    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 00:31:13.740051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:13.730552    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth" podUID="4a713f91-8d61-4d25-84d1-8121a511bd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 00:31:13.765053 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:13.760404    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 00:31:13.765053 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:13.760472    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj" podUID="c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 00:31:14.720313 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:14.720284    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"
Oct 30 00:31:14.745561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:14.745521    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 00:31:14.746152 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:14.746119    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth" podUID="4a713f91-8d61-4d25-84d1-8121a511bd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 00:31:14.761050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:14.758209    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"
Oct 30 00:31:14.765048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:14.761407    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 00:31:14.765048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:14.761469    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj" podUID="c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 00:31:14.780049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:14.779250    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth" podStartSLOduration=5.7792316889999995 podStartE2EDuration="5.779231689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:10.981985027 +0000 UTC m=+1580.507278905" watchObservedRunningTime="2026-10-30 00:31:14.779231689 +0000 UTC m=+1584.304525567"
Oct 30 00:31:15.744052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:15.739846    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 00:31:15.744052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:15.739932    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth" podUID="4a713f91-8d61-4d25-84d1-8121a511bd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 00:31:15.764830 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:15.764791    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 00:31:15.765464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:15.765434    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj" podUID="c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 00:31:16.652776 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.652737    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"]
Oct 30 00:31:16.658165 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.658139    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:16.684831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.684800    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"]
Oct 30 00:31:16.688962 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.688443    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-bc8b7d-serving-cert\""
Oct 30 00:31:16.708125 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.708087    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab92d406-7ea3-4706-bcb6-653e403cc882-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:16.708317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.708260    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jt55\" (UniqueName: \"kubernetes.io/projected/ab92d406-7ea3-4706-bcb6-653e403cc882-kube-api-access-6jt55\") pod \"maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:16.708466 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.708443    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-metrics-certs\") pod \"maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:16.708623 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.708605    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:16.729843 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.729226    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 00:31:16.729843 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.729302    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth" podUID="4a713f91-8d61-4d25-84d1-8121a511bd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 00:31:16.759621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.759583    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 00:31:16.759913 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.759863    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj" podUID="c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 00:31:16.809978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.809942    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-metrics-certs\") pod \"maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:16.810505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.810081    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:16.810505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.810142    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab92d406-7ea3-4706-bcb6-653e403cc882-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:16.810505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.810202    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jt55\" (UniqueName: \"kubernetes.io/projected/ab92d406-7ea3-4706-bcb6-653e403cc882-kube-api-access-6jt55\") pod \"maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:16.810988 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.810961    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab92d406-7ea3-4706-bcb6-653e403cc882-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:16.812645 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.812622    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-metrics-certs\") pod \"maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:16.813121 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.813098    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:16.832001 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.831954    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jt55\" (UniqueName: \"kubernetes.io/projected/ab92d406-7ea3-4706-bcb6-653e403cc882-kube-api-access-6jt55\") pod \"maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:16.994583 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:16.994557    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:17.327794 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:17.327760    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"]
Oct 30 00:31:17.330094 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:31:17.329959    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab92d406_7ea3_4706_bcb6_653e403cc882.slice/crio-80878389c70871e7f2a343b88630527aee9d023fdb2573a71f6ca096975d37d5 WatchSource:0}: Error finding container 80878389c70871e7f2a343b88630527aee9d023fdb2573a71f6ca096975d37d5: Status 404 returned error can't find the container with id 80878389c70871e7f2a343b88630527aee9d023fdb2573a71f6ca096975d37d5
Oct 30 00:31:17.333311 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:17.333284    4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:31:17.975724 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:17.975688    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt" event={"ID":"ab92d406-7ea3-4706-bcb6-653e403cc882","Type":"ContainerStarted","Data":"80878389c70871e7f2a343b88630527aee9d023fdb2573a71f6ca096975d37d5"}
Oct 30 00:31:18.295090 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:18.294046    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:18.295262 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:18.295249    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:18.299044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:18.297090    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:31:18.359973 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:18.359672    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:18.362112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:18.361112    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:18.367499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:18.367445    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:31:18.987806 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:18.987764    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt" event={"ID":"ab92d406-7ea3-4706-bcb6-653e403cc882","Type":"ContainerStarted","Data":"059a3afd22c9c3acf93ac2d89cbbda51dbef382fc65cdf94251335db985a7b16"}
Oct 30 00:31:18.988824 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:18.987866    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:19.034159 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:19.034102    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt" podStartSLOduration=2.510747575 podStartE2EDuration="3.034082142s" totalImagesPullingTime="523.334567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:19.028284747 +0000 UTC m=+1588.553578625" watchObservedRunningTime="2026-10-30 00:31:19.034082142 +0000 UTC m=+1588.559376020"
Oct 30 00:31:25.002931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:25.002903    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:31:33.904172 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:31:33.904101    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:31:48.120082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:48.119871    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:31:53.597177 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:53.597142    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"]
Oct 30 00:31:58.598131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:58.598076    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg" podUID="cf06d4e0-8bf5-441f-8e59-dd61fa7785f3" containerName="payload-pre-processing" containerID="cri-o://cb21530b471c8ea20acd35bdae90f34e5e80df1b78b636b0bad8f6f9b021f768" gracePeriod=25
Oct 30 00:31:58.664875 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:58.664845    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"]
Oct 30 00:31:58.665579 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:58.665500    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz" podUID="f15bdf08-77c7-4133-af8f-5dd59c119513" containerName="istio-proxy" containerID="cri-o://85b0af4ec03a0847e62a69206ba9d23588749f7e037440d9e3b5af3cefbef833" gracePeriod=30
Oct 30 00:31:58.703480 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:31:58.703452    4441 logging.go:55] [core] [Channel #2765 SubChannel #2766]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.142:50052", ServerName: "10.133.0.142:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.142:50052: connect: connection refused"
Oct 30 00:31:59.011169 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.011141    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"
Oct 30 00:31:59.016591 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.016565    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-plugins-config-volume\") pod \"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3\" (UID: \"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3\") "
Oct 30 00:31:59.016736 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.016657    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-kube-api-access-t5sqs\" (UniqueName: \"kubernetes.io/projected/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-kube-api-access-t5sqs\") pod \"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3\" (UID: \"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3\") "
Oct 30 00:31:59.016995 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.016971    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-plugins-config-volume" pod "cf06d4e0-8bf5-441f-8e59-dd61fa7785f3" (UID: "cf06d4e0-8bf5-441f-8e59-dd61fa7785f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:31:59.017386 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.017364    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:31:59.019276 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.019250    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-kube-api-access-t5sqs" pod "cf06d4e0-8bf5-441f-8e59-dd61fa7785f3" (UID: "cf06d4e0-8bf5-441f-8e59-dd61fa7785f3"). InnerVolumeSpecName "kube-api-access-t5sqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:31:59.118978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.118945    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5sqs\" (UniqueName: \"kubernetes.io/projected/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3-kube-api-access-t5sqs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:31:59.465040 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.464963    4441 generic.go:356] "Generic (PLEG): container finished" podID="cf06d4e0-8bf5-441f-8e59-dd61fa7785f3" containerID="cb21530b471c8ea20acd35bdae90f34e5e80df1b78b636b0bad8f6f9b021f768" exitCode=0
Oct 30 00:31:59.465040 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.465004    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"
Oct 30 00:31:59.465319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.465066    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg" event={"ID":"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3","Type":"ContainerDied","Data":"cb21530b471c8ea20acd35bdae90f34e5e80df1b78b636b0bad8f6f9b021f768"}
Oct 30 00:31:59.465319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.465150    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg" event={"ID":"cf06d4e0-8bf5-441f-8e59-dd61fa7785f3","Type":"ContainerDied","Data":"ac3de624db36b948f16b8b5ea6909c53718d9687055f192e2b46369ea1131666"}
Oct 30 00:31:59.465319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.465179    4441 scope.go:122] "RemoveContainer" containerID="cb21530b471c8ea20acd35bdae90f34e5e80df1b78b636b0bad8f6f9b021f768"
Oct 30 00:31:59.475441 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.475423    4441 scope.go:122] "RemoveContainer" containerID="cb21530b471c8ea20acd35bdae90f34e5e80df1b78b636b0bad8f6f9b021f768"
Oct 30 00:31:59.475706 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:31:59.475685    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb21530b471c8ea20acd35bdae90f34e5e80df1b78b636b0bad8f6f9b021f768\": container with ID starting with cb21530b471c8ea20acd35bdae90f34e5e80df1b78b636b0bad8f6f9b021f768 not found: ID does not exist" containerID="cb21530b471c8ea20acd35bdae90f34e5e80df1b78b636b0bad8f6f9b021f768"
Oct 30 00:31:59.475744 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.475715    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb21530b471c8ea20acd35bdae90f34e5e80df1b78b636b0bad8f6f9b021f768"} err="failed to get container status \"cb21530b471c8ea20acd35bdae90f34e5e80df1b78b636b0bad8f6f9b021f768\": rpc error: code = NotFound desc = could not find container \"cb21530b471c8ea20acd35bdae90f34e5e80df1b78b636b0bad8f6f9b021f768\": container with ID starting with cb21530b471c8ea20acd35bdae90f34e5e80df1b78b636b0bad8f6f9b021f768 not found: ID does not exist"
Oct 30 00:31:59.512927 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.512900    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"]
Oct 30 00:31:59.522539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.522516    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg"]
Oct 30 00:31:59.704012 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:31:59.703970    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1dd8c1-54dbdf6687-tb7wg" podUID="cf06d4e0-8bf5-441f-8e59-dd61fa7785f3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.142:50052\" within 1s: context deadline exceeded"
Oct 30 00:31:59.704607 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:31:59.704211    4441 logging.go:55] [core] [Channel #2765 SubChannel #2766]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.142:50052", ServerName: "10.133.0.142:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.142:50052: operation was canceled"
Oct 30 00:32:00.609978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:00.609946    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf06d4e0-8bf5-441f-8e59-dd61fa7785f3" path="/var/lib/kubelet/pods/cf06d4e0-8bf5-441f-8e59-dd61fa7785f3/volumes"
Oct 30 00:32:02.335432 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.335399    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"]
Oct 30 00:32:02.335915 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.335623    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz" podUID="8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24" containerName="maas-api" containerID="cri-o://c65ae642e9535e3d75454cf457e69f3fbe0e1480b69d5b734e221537d2c89f91" gracePeriod=30
Oct 30 00:32:02.502520 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.502487    4441 generic.go:356] "Generic (PLEG): container finished" podID="8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24" containerID="c65ae642e9535e3d75454cf457e69f3fbe0e1480b69d5b734e221537d2c89f91" exitCode=0
Oct 30 00:32:02.502793 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.502560    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz" event={"ID":"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24","Type":"ContainerDied","Data":"c65ae642e9535e3d75454cf457e69f3fbe0e1480b69d5b734e221537d2c89f91"}
Oct 30 00:32:02.750185 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.750161    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:32:02.804957 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.804910    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-maas-api-tls\") pod \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") "
Oct 30 00:32:02.805162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.805089    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-kube-api-access-gbmsh\" (UniqueName: \"kubernetes.io/projected/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-kube-api-access-gbmsh\") pod \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") "
Oct 30 00:32:02.805299 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.805279    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-metrics-certs\") pod \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") "
Oct 30 00:32:02.805711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.805399    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-openshift-service-ca\") pod \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\" (UID: \"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24\") "
Oct 30 00:32:02.806471 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.806439    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-openshift-service-ca" pod "8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24" (UID: "8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:02.807788 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.807761    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-maas-api-tls" pod "8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24" (UID: "8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:02.807922 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.807804    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-kube-api-access-gbmsh" pod "8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24" (UID: "8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24"). InnerVolumeSpecName "kube-api-access-gbmsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:02.808257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.808231    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-metrics-certs" pod "8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24" (UID: "8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:02.908847 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.908808    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbmsh\" (UniqueName: \"kubernetes.io/projected/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-kube-api-access-gbmsh\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:02.908847 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.908849    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:02.909132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.908864    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:02.909132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:02.908879    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:03.511569 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:03.511539    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"
Oct 30 00:32:03.512038 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:03.511566    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz" event={"ID":"8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24","Type":"ContainerDied","Data":"4527b3307764a4d89566855fe3efad638e3c0ef90ef4264f2611e5ec33a3b63d"}
Oct 30 00:32:03.512038 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:03.511629    4441 scope.go:122] "RemoveContainer" containerID="c65ae642e9535e3d75454cf457e69f3fbe0e1480b69d5b734e221537d2c89f91"
Oct 30 00:32:03.556498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:03.556473    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"]
Oct 30 00:32:03.566203 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:03.566178    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1dd8c1-9f84d5dfd-xshtz"]
Oct 30 00:32:03.912809 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:32:03.912765    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:32:03.971715 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:03.971688    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:32:04.033285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.033247    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-credential-socket\") pod \"f15bdf08-77c7-4133-af8f-5dd59c119513\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") "
Oct 30 00:32:04.033473 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.033317    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15bdf08-77c7-4133-af8f-5dd59c119513-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15bdf08-77c7-4133-af8f-5dd59c119513-istiod-ca-cert\") pod \"f15bdf08-77c7-4133-af8f-5dd59c119513\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") "
Oct 30 00:32:04.033473 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.033409    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-kube-api-access-8kbkb\" (UniqueName: \"kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-kube-api-access-8kbkb\") pod \"f15bdf08-77c7-4133-af8f-5dd59c119513\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") "
Oct 30 00:32:04.033473 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.033464    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-envoy\") pod \"f15bdf08-77c7-4133-af8f-5dd59c119513\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") "
Oct 30 00:32:04.033652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.033561    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-data\") pod \"f15bdf08-77c7-4133-af8f-5dd59c119513\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") "
Oct 30 00:32:04.033652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.033609    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-token\" (UniqueName: \"kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-token\") pod \"f15bdf08-77c7-4133-af8f-5dd59c119513\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") "
Oct 30 00:32:04.033904 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.033880    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-podinfo\") pod \"f15bdf08-77c7-4133-af8f-5dd59c119513\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") "
Oct 30 00:32:04.034008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.033924    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-socket\") pod \"f15bdf08-77c7-4133-af8f-5dd59c119513\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") "
Oct 30 00:32:04.034008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.033950    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-certs\") pod \"f15bdf08-77c7-4133-af8f-5dd59c119513\" (UID: \"f15bdf08-77c7-4133-af8f-5dd59c119513\") "
Oct 30 00:32:04.034891 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.034863    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-certs" pod "f15bdf08-77c7-4133-af8f-5dd59c119513" (UID: "f15bdf08-77c7-4133-af8f-5dd59c119513"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:04.036667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.035083    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-credential-socket" pod "f15bdf08-77c7-4133-af8f-5dd59c119513" (UID: "f15bdf08-77c7-4133-af8f-5dd59c119513"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:04.036667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.035629    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15bdf08-77c7-4133-af8f-5dd59c119513-istiod-ca-cert" pod "f15bdf08-77c7-4133-af8f-5dd59c119513" (UID: "f15bdf08-77c7-4133-af8f-5dd59c119513"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:04.036667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.035894    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-socket" pod "f15bdf08-77c7-4133-af8f-5dd59c119513" (UID: "f15bdf08-77c7-4133-af8f-5dd59c119513"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:04.036884 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.036329    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-data" pod "f15bdf08-77c7-4133-af8f-5dd59c119513" (UID: "f15bdf08-77c7-4133-af8f-5dd59c119513"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:04.043072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.041467    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-podinfo" pod "f15bdf08-77c7-4133-af8f-5dd59c119513" (UID: "f15bdf08-77c7-4133-af8f-5dd59c119513"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:32:04.043072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.041556    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-kube-api-access-8kbkb" pod "f15bdf08-77c7-4133-af8f-5dd59c119513" (UID: "f15bdf08-77c7-4133-af8f-5dd59c119513"). InnerVolumeSpecName "kube-api-access-8kbkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:04.043072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.041581    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-envoy" pod "f15bdf08-77c7-4133-af8f-5dd59c119513" (UID: "f15bdf08-77c7-4133-af8f-5dd59c119513"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:04.043072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.042422    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-token" pod "f15bdf08-77c7-4133-af8f-5dd59c119513" (UID: "f15bdf08-77c7-4133-af8f-5dd59c119513"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:04.135935 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.135895    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:04.135935 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.135928    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:04.135935 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.135940    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:04.136239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.135953    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:04.136239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.135966    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15bdf08-77c7-4133-af8f-5dd59c119513-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:04.136239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.135977    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kbkb\" (UniqueName: \"kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-kube-api-access-8kbkb\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:04.136239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.135988    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:04.136239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.135999    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:04.136239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.136009    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15bdf08-77c7-4133-af8f-5dd59c119513-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:04.521056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.520995    4441 generic.go:356] "Generic (PLEG): container finished" podID="f15bdf08-77c7-4133-af8f-5dd59c119513" containerID="85b0af4ec03a0847e62a69206ba9d23588749f7e037440d9e3b5af3cefbef833" exitCode=0
Oct 30 00:32:04.521056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.521056    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"
Oct 30 00:32:04.521531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.521066    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz" event={"ID":"f15bdf08-77c7-4133-af8f-5dd59c119513","Type":"ContainerDied","Data":"85b0af4ec03a0847e62a69206ba9d23588749f7e037440d9e3b5af3cefbef833"}
Oct 30 00:32:04.521531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.521126    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz" event={"ID":"f15bdf08-77c7-4133-af8f-5dd59c119513","Type":"ContainerDied","Data":"52f8183471f7f731bca737934d82a6ff13f7f2b9ff1911badfcc4a1b8ccf08fb"}
Oct 30 00:32:04.521531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.521156    4441 scope.go:122] "RemoveContainer" containerID="85b0af4ec03a0847e62a69206ba9d23588749f7e037440d9e3b5af3cefbef833"
Oct 30 00:32:04.533919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.533885    4441 scope.go:122] "RemoveContainer" containerID="85b0af4ec03a0847e62a69206ba9d23588749f7e037440d9e3b5af3cefbef833"
Oct 30 00:32:04.534282 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:32:04.534212    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b0af4ec03a0847e62a69206ba9d23588749f7e037440d9e3b5af3cefbef833\": container with ID starting with 85b0af4ec03a0847e62a69206ba9d23588749f7e037440d9e3b5af3cefbef833 not found: ID does not exist" containerID="85b0af4ec03a0847e62a69206ba9d23588749f7e037440d9e3b5af3cefbef833"
Oct 30 00:32:04.534282 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.534250    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b0af4ec03a0847e62a69206ba9d23588749f7e037440d9e3b5af3cefbef833"} err="failed to get container status \"85b0af4ec03a0847e62a69206ba9d23588749f7e037440d9e3b5af3cefbef833\": rpc error: code = NotFound desc = could not find container \"85b0af4ec03a0847e62a69206ba9d23588749f7e037440d9e3b5af3cefbef833\": container with ID starting with 85b0af4ec03a0847e62a69206ba9d23588749f7e037440d9e3b5af3cefbef833 not found: ID does not exist"
Oct 30 00:32:04.568922 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.568890    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"]
Oct 30 00:32:04.578082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.578052    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-8sjfz"]
Oct 30 00:32:04.613925 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.613893    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24" path="/var/lib/kubelet/pods/8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24/volumes"
Oct 30 00:32:04.614647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:04.614617    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15bdf08-77c7-4133-af8f-5dd59c119513" path="/var/lib/kubelet/pods/f15bdf08-77c7-4133-af8f-5dd59c119513/volumes"
Oct 30 00:32:07.194464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.194431    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"]
Oct 30 00:32:07.195140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.195125    4441 cpu_manager.go:414] "containerMap: removing container" podUID="8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24" containerName="maas-api"
Oct 30 00:32:07.195197 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.195141    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24" containerName="maas-api"
Oct 30 00:32:07.195197 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.195153    4441 cpu_manager.go:414] "containerMap: removing container" podUID="cf06d4e0-8bf5-441f-8e59-dd61fa7785f3" containerName="payload-pre-processing"
Oct 30 00:32:07.195197 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.195158    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf06d4e0-8bf5-441f-8e59-dd61fa7785f3" containerName="payload-pre-processing"
Oct 30 00:32:07.195197 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.195181    4441 cpu_manager.go:414] "containerMap: removing container" podUID="f15bdf08-77c7-4133-af8f-5dd59c119513" containerName="istio-proxy"
Oct 30 00:32:07.195197 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.195185    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15bdf08-77c7-4133-af8f-5dd59c119513" containerName="istio-proxy"
Oct 30 00:32:07.195352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.195277    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec9b2fa-26ca-4cd6-877b-e5a94fa9ce24" containerName="maas-api"
Oct 30 00:32:07.195352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.195285    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf06d4e0-8bf5-441f-8e59-dd61fa7785f3" containerName="payload-pre-processing"
Oct 30 00:32:07.195352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.195295    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15bdf08-77c7-4133-af8f-5dd59c119513" containerName="istio-proxy"
Oct 30 00:32:07.200344 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.200319    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.228994 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.228965    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"]
Oct 30 00:32:07.292851 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.292807    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.293062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.292862    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.293146 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.293119    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvwp\" (UniqueName: \"kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-kube-api-access-bxvwp\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.293285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.293260    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.293356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.293323    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.293448 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.293393    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.293522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.293503    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.293590 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.293571    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.293643 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.293633    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.394690 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.394654    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvwp\" (UniqueName: \"kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-kube-api-access-bxvwp\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.394877 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.394797    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.394877 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.394816    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.394877 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.394856    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.395077 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.394934    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.395077 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.394969    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.395077 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.395040    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.395252 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.395104    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.395252 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.395134    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.395367 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.395315    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.395453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.395415    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.395507 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.395470    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.395507 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.395492    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.395942 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.395918    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.397355 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.397333    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.397554 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.397531    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.415319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.415298    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.415476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.415459    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvwp\" (UniqueName: \"kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-kube-api-access-bxvwp\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.515376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.515350    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:07.874670 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.874640    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"]
Oct 30 00:32:07.877833 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:32:07.877802    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4141111_103c_47f4_ac95_1b0d19fbd5cd.slice/crio-542c729268f881d59cb861204db9c27eeffa8db98696cfa8fd60e6a1d8cd55ce WatchSource:0}: Error finding container 542c729268f881d59cb861204db9c27eeffa8db98696cfa8fd60e6a1d8cd55ce: Status 404 returned error can't find the container with id 542c729268f881d59cb861204db9c27eeffa8db98696cfa8fd60e6a1d8cd55ce
Oct 30 00:32:07.881044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.880984    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:32:07.881153 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.881097    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:32:07.881153 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:07.881147    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:32:08.566396 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:08.566359    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q" event={"ID":"d4141111-103c-47f4-ac95-1b0d19fbd5cd","Type":"ContainerStarted","Data":"b43ff405c64139920fc694facc5d0d099e89026dd1f98a1f8abc29b45dbd5520"}
Oct 30 00:32:08.566938 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:08.566428    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q" event={"ID":"d4141111-103c-47f4-ac95-1b0d19fbd5cd","Type":"ContainerStarted","Data":"542c729268f881d59cb861204db9c27eeffa8db98696cfa8fd60e6a1d8cd55ce"}
Oct 30 00:32:08.635049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:08.627850    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q" podStartSLOduration=1.627831399 podStartE2EDuration="1.627831399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:08.616613483 +0000 UTC m=+1638.141907361" watchObservedRunningTime="2026-10-30 00:32:08.627831399 +0000 UTC m=+1638.153125276"
Oct 30 00:32:09.516313 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:09.516270    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:09.518213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:09.518185    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 00:32:09.518338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:09.518236    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q" podUID="d4141111-103c-47f4-ac95-1b0d19fbd5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 00:32:09.670753 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:09.670699    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"]
Oct 30 00:32:10.528050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:10.524400    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 00:32:10.528050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:10.524476    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q" podUID="d4141111-103c-47f4-ac95-1b0d19fbd5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 00:32:11.516091 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:11.516015    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 00:32:11.516517 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:11.516111    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q" podUID="d4141111-103c-47f4-ac95-1b0d19fbd5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 00:32:12.515968 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:12.515930    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 00:32:12.516214 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:12.515986    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q" podUID="d4141111-103c-47f4-ac95-1b0d19fbd5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 00:32:13.516772 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:13.516731    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 00:32:13.517281 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:13.516815    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q" podUID="d4141111-103c-47f4-ac95-1b0d19fbd5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 00:32:14.516341 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:14.516302    4441 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 00:32:14.516566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:14.516409    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q" podUID="d4141111-103c-47f4-ac95-1b0d19fbd5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 00:32:14.672794 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:14.672757    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4" podUID="f80a0aed-06cb-4ca3-a9e4-142a932076fe" containerName="payload-processing" containerID="cri-o://9891cb39a655643f9f371e44b32372531e010f4d77d06d10ea95619ddaf891fe" gracePeriod=25
Oct 30 00:32:15.163045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.162675    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"
Oct 30 00:32:15.313657 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.313624    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f80a0aed-06cb-4ca3-a9e4-142a932076fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80a0aed-06cb-4ca3-a9e4-142a932076fe-plugins-config-volume\") pod \"f80a0aed-06cb-4ca3-a9e4-142a932076fe\" (UID: \"f80a0aed-06cb-4ca3-a9e4-142a932076fe\") "
Oct 30 00:32:15.313852 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.313747    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80a0aed-06cb-4ca3-a9e4-142a932076fe-kube-api-access-xvzc7\" (UniqueName: \"kubernetes.io/projected/f80a0aed-06cb-4ca3-a9e4-142a932076fe-kube-api-access-xvzc7\") pod \"f80a0aed-06cb-4ca3-a9e4-142a932076fe\" (UID: \"f80a0aed-06cb-4ca3-a9e4-142a932076fe\") "
Oct 30 00:32:15.314046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.313998    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80a0aed-06cb-4ca3-a9e4-142a932076fe-plugins-config-volume" pod "f80a0aed-06cb-4ca3-a9e4-142a932076fe" (UID: "f80a0aed-06cb-4ca3-a9e4-142a932076fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:15.314423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.314403    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80a0aed-06cb-4ca3-a9e4-142a932076fe-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:15.316175 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.316152    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80a0aed-06cb-4ca3-a9e4-142a932076fe-kube-api-access-xvzc7" pod "f80a0aed-06cb-4ca3-a9e4-142a932076fe" (UID: "f80a0aed-06cb-4ca3-a9e4-142a932076fe"). InnerVolumeSpecName "kube-api-access-xvzc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:15.415866 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.415793    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvzc7\" (UniqueName: \"kubernetes.io/projected/f80a0aed-06cb-4ca3-a9e4-142a932076fe-kube-api-access-xvzc7\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:15.520274 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.520241    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:15.520701 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.520681    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:15.521441 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.521412    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:32:15.649146 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.649098    4441 generic.go:356] "Generic (PLEG): container finished" podID="f80a0aed-06cb-4ca3-a9e4-142a932076fe" containerID="9891cb39a655643f9f371e44b32372531e010f4d77d06d10ea95619ddaf891fe" exitCode=0
Oct 30 00:32:15.650130 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.650108    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"
Oct 30 00:32:15.653223 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.653192    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4" event={"ID":"f80a0aed-06cb-4ca3-a9e4-142a932076fe","Type":"ContainerDied","Data":"9891cb39a655643f9f371e44b32372531e010f4d77d06d10ea95619ddaf891fe"}
Oct 30 00:32:15.653341 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.653256    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4" event={"ID":"f80a0aed-06cb-4ca3-a9e4-142a932076fe","Type":"ContainerDied","Data":"50c688ccf4a5ada0bf5bb552d2d25d5e4c62b74bf8c913d59dac05045ac65a82"}
Oct 30 00:32:15.653341 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.653287    4441 scope.go:122] "RemoveContainer" containerID="9891cb39a655643f9f371e44b32372531e010f4d77d06d10ea95619ddaf891fe"
Oct 30 00:32:15.672164 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.672145    4441 scope.go:122] "RemoveContainer" containerID="9891cb39a655643f9f371e44b32372531e010f4d77d06d10ea95619ddaf891fe"
Oct 30 00:32:15.672480 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:32:15.672454    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9891cb39a655643f9f371e44b32372531e010f4d77d06d10ea95619ddaf891fe\": container with ID starting with 9891cb39a655643f9f371e44b32372531e010f4d77d06d10ea95619ddaf891fe not found: ID does not exist" containerID="9891cb39a655643f9f371e44b32372531e010f4d77d06d10ea95619ddaf891fe"
Oct 30 00:32:15.672549 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.672494    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9891cb39a655643f9f371e44b32372531e010f4d77d06d10ea95619ddaf891fe"} err="failed to get container status \"9891cb39a655643f9f371e44b32372531e010f4d77d06d10ea95619ddaf891fe\": rpc error: code = NotFound desc = could not find container \"9891cb39a655643f9f371e44b32372531e010f4d77d06d10ea95619ddaf891fe\": container with ID starting with 9891cb39a655643f9f371e44b32372531e010f4d77d06d10ea95619ddaf891fe not found: ID does not exist"
Oct 30 00:32:15.701793 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.701759    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"]
Oct 30 00:32:15.703168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:15.703149    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3e13d-6c65cbb88b-rnmj4"]
Oct 30 00:32:16.662047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:16.644867    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f80a0aed-06cb-4ca3-a9e4-142a932076fe" path="/var/lib/kubelet/pods/f80a0aed-06cb-4ca3-a9e4-142a932076fe/volumes"
Oct 30 00:32:21.223971 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:21.223931    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"]
Oct 30 00:32:21.256580 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:21.256546    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"]
Oct 30 00:32:24.353103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:24.352892    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"]
Oct 30 00:32:24.355216 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:24.353937    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt" podUID="ab92d406-7ea3-4706-bcb6-653e403cc882" containerName="maas-api" containerID="cri-o://059a3afd22c9c3acf93ac2d89cbbda51dbef382fc65cdf94251335db985a7b16" gracePeriod=30
Oct 30 00:32:24.783154 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:24.780264    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:32:24.913706 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:24.906539    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-maas-api-tls\") pod \"ab92d406-7ea3-4706-bcb6-653e403cc882\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") "
Oct 30 00:32:24.914305 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:24.914205    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab92d406-7ea3-4706-bcb6-653e403cc882-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab92d406-7ea3-4706-bcb6-653e403cc882-openshift-service-ca\") pod \"ab92d406-7ea3-4706-bcb6-653e403cc882\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") "
Oct 30 00:32:24.914569 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:24.914548    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab92d406-7ea3-4706-bcb6-653e403cc882-kube-api-access-6jt55\" (UniqueName: \"kubernetes.io/projected/ab92d406-7ea3-4706-bcb6-653e403cc882-kube-api-access-6jt55\") pod \"ab92d406-7ea3-4706-bcb6-653e403cc882\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") "
Oct 30 00:32:24.914999 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:24.914956    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-metrics-certs\") pod \"ab92d406-7ea3-4706-bcb6-653e403cc882\" (UID: \"ab92d406-7ea3-4706-bcb6-653e403cc882\") "
Oct 30 00:32:24.929454 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:24.929423    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-metrics-certs" pod "ab92d406-7ea3-4706-bcb6-653e403cc882" (UID: "ab92d406-7ea3-4706-bcb6-653e403cc882"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:24.929815 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:24.929768    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab92d406-7ea3-4706-bcb6-653e403cc882-kube-api-access-6jt55" pod "ab92d406-7ea3-4706-bcb6-653e403cc882" (UID: "ab92d406-7ea3-4706-bcb6-653e403cc882"). InnerVolumeSpecName "kube-api-access-6jt55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:24.943292 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:24.942248    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab92d406-7ea3-4706-bcb6-653e403cc882-openshift-service-ca" pod "ab92d406-7ea3-4706-bcb6-653e403cc882" (UID: "ab92d406-7ea3-4706-bcb6-653e403cc882"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:24.943292 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:24.942632    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-maas-api-tls" pod "ab92d406-7ea3-4706-bcb6-653e403cc882" (UID: "ab92d406-7ea3-4706-bcb6-653e403cc882"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:25.002440 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:25.002346    4441 generic.go:356] "Generic (PLEG): container finished" podID="ab92d406-7ea3-4706-bcb6-653e403cc882" containerID="059a3afd22c9c3acf93ac2d89cbbda51dbef382fc65cdf94251335db985a7b16" exitCode=0
Oct 30 00:32:25.002656 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:25.002629    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt" event={"ID":"ab92d406-7ea3-4706-bcb6-653e403cc882","Type":"ContainerDied","Data":"059a3afd22c9c3acf93ac2d89cbbda51dbef382fc65cdf94251335db985a7b16"}
Oct 30 00:32:25.002731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:25.002692    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt" event={"ID":"ab92d406-7ea3-4706-bcb6-653e403cc882","Type":"ContainerDied","Data":"80878389c70871e7f2a343b88630527aee9d023fdb2573a71f6ca096975d37d5"}
Oct 30 00:32:25.002731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:25.002724    4441 scope.go:122] "RemoveContainer" containerID="059a3afd22c9c3acf93ac2d89cbbda51dbef382fc65cdf94251335db985a7b16"
Oct 30 00:32:25.003566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:25.002911    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"
Oct 30 00:32:25.017957 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:25.017929    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab92d406-7ea3-4706-bcb6-653e403cc882-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:25.018130 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:25.018113    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jt55\" (UniqueName: \"kubernetes.io/projected/ab92d406-7ea3-4706-bcb6-653e403cc882-kube-api-access-6jt55\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:25.018245 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:25.018233    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:25.018332 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:25.018322    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab92d406-7ea3-4706-bcb6-653e403cc882-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:25.023419 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:25.023400    4441 scope.go:122] "RemoveContainer" containerID="059a3afd22c9c3acf93ac2d89cbbda51dbef382fc65cdf94251335db985a7b16"
Oct 30 00:32:25.023769 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:32:25.023747    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059a3afd22c9c3acf93ac2d89cbbda51dbef382fc65cdf94251335db985a7b16\": container with ID starting with 059a3afd22c9c3acf93ac2d89cbbda51dbef382fc65cdf94251335db985a7b16 not found: ID does not exist" containerID="059a3afd22c9c3acf93ac2d89cbbda51dbef382fc65cdf94251335db985a7b16"
Oct 30 00:32:25.023867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:25.023776    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059a3afd22c9c3acf93ac2d89cbbda51dbef382fc65cdf94251335db985a7b16"} err="failed to get container status \"059a3afd22c9c3acf93ac2d89cbbda51dbef382fc65cdf94251335db985a7b16\": rpc error: code = NotFound desc = could not find container \"059a3afd22c9c3acf93ac2d89cbbda51dbef382fc65cdf94251335db985a7b16\": container with ID starting with 059a3afd22c9c3acf93ac2d89cbbda51dbef382fc65cdf94251335db985a7b16 not found: ID does not exist"
Oct 30 00:32:25.055114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:25.055083    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"]
Oct 30 00:32:25.058680 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:25.058654    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bc8b7d-7666ccfdc8-mbqxt"]
Oct 30 00:32:26.224985 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.224936    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm" podUID="98b69041-6f23-4680-a25f-2d38b2228ed3" containerName="payload-processing" containerID="cri-o://033aad24f18a74d3eb52e82df3e2eb633d534b978c0d59ef0fe82dcfb3afa38f" gracePeriod=25
Oct 30 00:32:26.257895 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.257811    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5" podUID="8030ad10-9610-4159-a607-92ca15758e63" containerName="payload-pre-processing" containerID="cri-o://c758c6dc5f1f08396c20a0a94d2f4a6067f9b036df7fab9f660162e98024993e" gracePeriod=25
Oct 30 00:32:26.563551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.563518    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"
Oct 30 00:32:26.614045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.614003    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab92d406-7ea3-4706-bcb6-653e403cc882" path="/var/lib/kubelet/pods/ab92d406-7ea3-4706-bcb6-653e403cc882/volumes"
Oct 30 00:32:26.616238 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.616210    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"
Oct 30 00:32:26.666417 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.666386    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b69041-6f23-4680-a25f-2d38b2228ed3-kube-api-access-9cwhm\" (UniqueName: \"kubernetes.io/projected/98b69041-6f23-4680-a25f-2d38b2228ed3-kube-api-access-9cwhm\") pod \"98b69041-6f23-4680-a25f-2d38b2228ed3\" (UID: \"98b69041-6f23-4680-a25f-2d38b2228ed3\") "
Oct 30 00:32:26.667285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.667261    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8030ad10-9610-4159-a607-92ca15758e63-kube-api-access-skl89\" (UniqueName: \"kubernetes.io/projected/8030ad10-9610-4159-a607-92ca15758e63-kube-api-access-skl89\") pod \"8030ad10-9610-4159-a607-92ca15758e63\" (UID: \"8030ad10-9610-4159-a607-92ca15758e63\") "
Oct 30 00:32:26.667561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.667541    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8030ad10-9610-4159-a607-92ca15758e63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8030ad10-9610-4159-a607-92ca15758e63-plugins-config-volume\") pod \"8030ad10-9610-4159-a607-92ca15758e63\" (UID: \"8030ad10-9610-4159-a607-92ca15758e63\") "
Oct 30 00:32:26.667821 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.667801    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98b69041-6f23-4680-a25f-2d38b2228ed3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98b69041-6f23-4680-a25f-2d38b2228ed3-plugins-config-volume\") pod \"98b69041-6f23-4680-a25f-2d38b2228ed3\" (UID: \"98b69041-6f23-4680-a25f-2d38b2228ed3\") "
Oct 30 00:32:26.669834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.668382    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b69041-6f23-4680-a25f-2d38b2228ed3-plugins-config-volume" pod "98b69041-6f23-4680-a25f-2d38b2228ed3" (UID: "98b69041-6f23-4680-a25f-2d38b2228ed3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:26.671900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.668384    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8030ad10-9610-4159-a607-92ca15758e63-plugins-config-volume" pod "8030ad10-9610-4159-a607-92ca15758e63" (UID: "8030ad10-9610-4159-a607-92ca15758e63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:26.671900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.671617    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8030ad10-9610-4159-a607-92ca15758e63-kube-api-access-skl89" pod "8030ad10-9610-4159-a607-92ca15758e63" (UID: "8030ad10-9610-4159-a607-92ca15758e63"). InnerVolumeSpecName "kube-api-access-skl89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:26.674635 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.674613    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b69041-6f23-4680-a25f-2d38b2228ed3-kube-api-access-9cwhm" pod "98b69041-6f23-4680-a25f-2d38b2228ed3" (UID: "98b69041-6f23-4680-a25f-2d38b2228ed3"). InnerVolumeSpecName "kube-api-access-9cwhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:26.775624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.775578    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98b69041-6f23-4680-a25f-2d38b2228ed3-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:26.775624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.775622    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cwhm\" (UniqueName: \"kubernetes.io/projected/98b69041-6f23-4680-a25f-2d38b2228ed3-kube-api-access-9cwhm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:26.775906 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.775638    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skl89\" (UniqueName: \"kubernetes.io/projected/8030ad10-9610-4159-a607-92ca15758e63-kube-api-access-skl89\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:26.775906 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:26.775652    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8030ad10-9610-4159-a607-92ca15758e63-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:27.018064 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.017988    4441 generic.go:356] "Generic (PLEG): container finished" podID="8030ad10-9610-4159-a607-92ca15758e63" containerID="c758c6dc5f1f08396c20a0a94d2f4a6067f9b036df7fab9f660162e98024993e" exitCode=0
Oct 30 00:32:27.018064 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.018053    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"
Oct 30 00:32:27.018064 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.018058    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5" event={"ID":"8030ad10-9610-4159-a607-92ca15758e63","Type":"ContainerDied","Data":"c758c6dc5f1f08396c20a0a94d2f4a6067f9b036df7fab9f660162e98024993e"}
Oct 30 00:32:27.018344 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.018127    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5" event={"ID":"8030ad10-9610-4159-a607-92ca15758e63","Type":"ContainerDied","Data":"87ae576048388d8aa216b94366cce6c8a18ed59b299035df5f2ad2f960b36383"}
Oct 30 00:32:27.018344 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.018155    4441 scope.go:122] "RemoveContainer" containerID="c758c6dc5f1f08396c20a0a94d2f4a6067f9b036df7fab9f660162e98024993e"
Oct 30 00:32:27.019717 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.019695    4441 generic.go:356] "Generic (PLEG): container finished" podID="98b69041-6f23-4680-a25f-2d38b2228ed3" containerID="033aad24f18a74d3eb52e82df3e2eb633d534b978c0d59ef0fe82dcfb3afa38f" exitCode=0
Oct 30 00:32:27.019846 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.019745    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"
Oct 30 00:32:27.019846 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.019778    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm" event={"ID":"98b69041-6f23-4680-a25f-2d38b2228ed3","Type":"ContainerDied","Data":"033aad24f18a74d3eb52e82df3e2eb633d534b978c0d59ef0fe82dcfb3afa38f"}
Oct 30 00:32:27.019846 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.019830    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm" event={"ID":"98b69041-6f23-4680-a25f-2d38b2228ed3","Type":"ContainerDied","Data":"2c48d265073167b63fa0278a9d3b226e94606dff86f0dc78d55d00c02f3ecb26"}
Oct 30 00:32:27.031333 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.031177    4441 scope.go:122] "RemoveContainer" containerID="c758c6dc5f1f08396c20a0a94d2f4a6067f9b036df7fab9f660162e98024993e"
Oct 30 00:32:27.031756 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:32:27.031735    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c758c6dc5f1f08396c20a0a94d2f4a6067f9b036df7fab9f660162e98024993e\": container with ID starting with c758c6dc5f1f08396c20a0a94d2f4a6067f9b036df7fab9f660162e98024993e not found: ID does not exist" containerID="c758c6dc5f1f08396c20a0a94d2f4a6067f9b036df7fab9f660162e98024993e"
Oct 30 00:32:27.031869 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.031763    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c758c6dc5f1f08396c20a0a94d2f4a6067f9b036df7fab9f660162e98024993e"} err="failed to get container status \"c758c6dc5f1f08396c20a0a94d2f4a6067f9b036df7fab9f660162e98024993e\": rpc error: code = NotFound desc = could not find container \"c758c6dc5f1f08396c20a0a94d2f4a6067f9b036df7fab9f660162e98024993e\": container with ID starting with c758c6dc5f1f08396c20a0a94d2f4a6067f9b036df7fab9f660162e98024993e not found: ID does not exist"
Oct 30 00:32:27.031869 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.031781    4441 scope.go:122] "RemoveContainer" containerID="033aad24f18a74d3eb52e82df3e2eb633d534b978c0d59ef0fe82dcfb3afa38f"
Oct 30 00:32:27.042964 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.042946    4441 scope.go:122] "RemoveContainer" containerID="033aad24f18a74d3eb52e82df3e2eb633d534b978c0d59ef0fe82dcfb3afa38f"
Oct 30 00:32:27.043271 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:32:27.043251    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033aad24f18a74d3eb52e82df3e2eb633d534b978c0d59ef0fe82dcfb3afa38f\": container with ID starting with 033aad24f18a74d3eb52e82df3e2eb633d534b978c0d59ef0fe82dcfb3afa38f not found: ID does not exist" containerID="033aad24f18a74d3eb52e82df3e2eb633d534b978c0d59ef0fe82dcfb3afa38f"
Oct 30 00:32:27.043363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.043278    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033aad24f18a74d3eb52e82df3e2eb633d534b978c0d59ef0fe82dcfb3afa38f"} err="failed to get container status \"033aad24f18a74d3eb52e82df3e2eb633d534b978c0d59ef0fe82dcfb3afa38f\": rpc error: code = NotFound desc = could not find container \"033aad24f18a74d3eb52e82df3e2eb633d534b978c0d59ef0fe82dcfb3afa38f\": container with ID starting with 033aad24f18a74d3eb52e82df3e2eb633d534b978c0d59ef0fe82dcfb3afa38f not found: ID does not exist"
Oct 30 00:32:27.076432 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.076411    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"]
Oct 30 00:32:27.085617 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.085599    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bc8b7d-58c94ccfc5-qfnh5"]
Oct 30 00:32:27.107739 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.107681    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"]
Oct 30 00:32:27.110619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:27.110600    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bc8b7d-69b44c9cd-vdphm"]
Oct 30 00:32:28.610779 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:28.610749    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8030ad10-9610-4159-a607-92ca15758e63" path="/var/lib/kubelet/pods/8030ad10-9610-4159-a607-92ca15758e63/volumes"
Oct 30 00:32:28.611247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:28.611227    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98b69041-6f23-4680-a25f-2d38b2228ed3" path="/var/lib/kubelet/pods/98b69041-6f23-4680-a25f-2d38b2228ed3/volumes"
Oct 30 00:32:32.813995 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:32.813958    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"]
Oct 30 00:32:32.814584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:32.814211    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx" podUID="42d7f0fa-9352-4408-b1a1-cc80af286710" containerName="istio-proxy" containerID="cri-o://cfe8b748064f61fba58bd2514ba5c22491f9aa5cb715fd5ed49dd90975e18d56" gracePeriod=30
Oct 30 00:32:33.905568 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:32:33.905528    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:32:38.112895 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.112872    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:32:38.129874 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.129849    4441 generic.go:356] "Generic (PLEG): container finished" podID="42d7f0fa-9352-4408-b1a1-cc80af286710" containerID="cfe8b748064f61fba58bd2514ba5c22491f9aa5cb715fd5ed49dd90975e18d56" exitCode=0
Oct 30 00:32:38.129987 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.129922    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx" event={"ID":"42d7f0fa-9352-4408-b1a1-cc80af286710","Type":"ContainerDied","Data":"cfe8b748064f61fba58bd2514ba5c22491f9aa5cb715fd5ed49dd90975e18d56"}
Oct 30 00:32:38.129987 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.129962    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx" event={"ID":"42d7f0fa-9352-4408-b1a1-cc80af286710","Type":"ContainerDied","Data":"2f2fe6e6fcabefc97277d9f08bbd73d71223a90763faa0f5014776809bea248b"}
Oct 30 00:32:38.130105 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.129986    4441 scope.go:122] "RemoveContainer" containerID="cfe8b748064f61fba58bd2514ba5c22491f9aa5cb715fd5ed49dd90975e18d56"
Oct 30 00:32:38.130146 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.130127    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"
Oct 30 00:32:38.141664 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.141640    4441 scope.go:122] "RemoveContainer" containerID="cfe8b748064f61fba58bd2514ba5c22491f9aa5cb715fd5ed49dd90975e18d56"
Oct 30 00:32:38.141927 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:32:38.141911    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe8b748064f61fba58bd2514ba5c22491f9aa5cb715fd5ed49dd90975e18d56\": container with ID starting with cfe8b748064f61fba58bd2514ba5c22491f9aa5cb715fd5ed49dd90975e18d56 not found: ID does not exist" containerID="cfe8b748064f61fba58bd2514ba5c22491f9aa5cb715fd5ed49dd90975e18d56"
Oct 30 00:32:38.141980 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.141933    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe8b748064f61fba58bd2514ba5c22491f9aa5cb715fd5ed49dd90975e18d56"} err="failed to get container status \"cfe8b748064f61fba58bd2514ba5c22491f9aa5cb715fd5ed49dd90975e18d56\": rpc error: code = NotFound desc = could not find container \"cfe8b748064f61fba58bd2514ba5c22491f9aa5cb715fd5ed49dd90975e18d56\": container with ID starting with cfe8b748064f61fba58bd2514ba5c22491f9aa5cb715fd5ed49dd90975e18d56 not found: ID does not exist"
Oct 30 00:32:38.246703 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.246665    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-socket\") pod \"42d7f0fa-9352-4408-b1a1-cc80af286710\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") "
Oct 30 00:32:38.246896 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.246779    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-certs\") pod \"42d7f0fa-9352-4408-b1a1-cc80af286710\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") "
Oct 30 00:32:38.246896 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.246809    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d7f0fa-9352-4408-b1a1-cc80af286710-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d7f0fa-9352-4408-b1a1-cc80af286710-istiod-ca-cert\") pod \"42d7f0fa-9352-4408-b1a1-cc80af286710\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") "
Oct 30 00:32:38.247015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.246910    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-token\" (UniqueName: \"kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-token\") pod \"42d7f0fa-9352-4408-b1a1-cc80af286710\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") "
Oct 30 00:32:38.247015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.246919    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-socket" pod "42d7f0fa-9352-4408-b1a1-cc80af286710" (UID: "42d7f0fa-9352-4408-b1a1-cc80af286710"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:38.247015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.246966    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-envoy\") pod \"42d7f0fa-9352-4408-b1a1-cc80af286710\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") "
Oct 30 00:32:38.247220 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.247062    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-podinfo\") pod \"42d7f0fa-9352-4408-b1a1-cc80af286710\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") "
Oct 30 00:32:38.247220 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.247107    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-data\") pod \"42d7f0fa-9352-4408-b1a1-cc80af286710\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") "
Oct 30 00:32:38.247220 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.247115    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-certs" pod "42d7f0fa-9352-4408-b1a1-cc80af286710" (UID: "42d7f0fa-9352-4408-b1a1-cc80af286710"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:38.247220 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.247194    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-credential-socket\") pod \"42d7f0fa-9352-4408-b1a1-cc80af286710\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") "
Oct 30 00:32:38.247423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.247238    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-kube-api-access-fs7lf\" (UniqueName: \"kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-kube-api-access-fs7lf\") pod \"42d7f0fa-9352-4408-b1a1-cc80af286710\" (UID: \"42d7f0fa-9352-4408-b1a1-cc80af286710\") "
Oct 30 00:32:38.247423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.247262    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d7f0fa-9352-4408-b1a1-cc80af286710-istiod-ca-cert" pod "42d7f0fa-9352-4408-b1a1-cc80af286710" (UID: "42d7f0fa-9352-4408-b1a1-cc80af286710"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:38.247525 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.247431    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-data" pod "42d7f0fa-9352-4408-b1a1-cc80af286710" (UID: "42d7f0fa-9352-4408-b1a1-cc80af286710"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:38.249576 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.249549    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:38.249732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.249580    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:38.249732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.249604    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d7f0fa-9352-4408-b1a1-cc80af286710-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:38.249732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.249618    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:38.249732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.247537    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-credential-socket" pod "42d7f0fa-9352-4408-b1a1-cc80af286710" (UID: "42d7f0fa-9352-4408-b1a1-cc80af286710"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:38.252486 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.252455    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-podinfo" pod "42d7f0fa-9352-4408-b1a1-cc80af286710" (UID: "42d7f0fa-9352-4408-b1a1-cc80af286710"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:32:38.261486 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.261462    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-token" pod "42d7f0fa-9352-4408-b1a1-cc80af286710" (UID: "42d7f0fa-9352-4408-b1a1-cc80af286710"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:38.265774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.265753    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-kube-api-access-fs7lf" pod "42d7f0fa-9352-4408-b1a1-cc80af286710" (UID: "42d7f0fa-9352-4408-b1a1-cc80af286710"). InnerVolumeSpecName "kube-api-access-fs7lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:38.266011 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.265756    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-envoy" pod "42d7f0fa-9352-4408-b1a1-cc80af286710" (UID: "42d7f0fa-9352-4408-b1a1-cc80af286710"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:38.351538 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.351453    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:38.351538 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.351493    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:38.351538 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.351508    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs7lf\" (UniqueName: \"kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-kube-api-access-fs7lf\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:38.351538 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.351522    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:38.351538 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.351535    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d7f0fa-9352-4408-b1a1-cc80af286710-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:38.474942 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.474908    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"]
Oct 30 00:32:38.478350 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.478324    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-nwldx"]
Oct 30 00:32:38.609674 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:38.609602    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d7f0fa-9352-4408-b1a1-cc80af286710" path="/var/lib/kubelet/pods/42d7f0fa-9352-4408-b1a1-cc80af286710/volumes"
Oct 30 00:32:47.805331 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:47.805277    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"]
Oct 30 00:32:47.805881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:47.805476    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw" podUID="4489355f-aad2-4394-bd10-7a78024d6458" containerName="istio-proxy" containerID="cri-o://9ec72f9547d83ad4ee92e17fd4e5e961e148b98f9ac2cafc80a6202423eecd11" gracePeriod=30
Oct 30 00:32:48.440054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.432078    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"]
Oct 30 00:32:48.440054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.432326    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k" podUID="192d192e-cab7-42b8-86d3-335abcced580" containerName="discovery" containerID="cri-o://eb7b71f1494879e229da0246d7e694c024ae81aa8386a3d51f273df0dddefe8c" gracePeriod=30
Oct 30 00:32:48.789460 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.789434    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:32:48.890330 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.890298    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-ca-configmap\") pod \"192d192e-cab7-42b8-86d3-335abcced580\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") "
Oct 30 00:32:48.890799 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.890347    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-cacerts\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-cacerts\") pod \"192d192e-cab7-42b8-86d3-335abcced580\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") "
Oct 30 00:32:48.890799 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.890466    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-istio-token\" (UniqueName: \"kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-istio-token\") pod \"192d192e-cab7-42b8-86d3-335abcced580\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") "
Oct 30 00:32:48.890799 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.890545    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-kubeconfig\") pod \"192d192e-cab7-42b8-86d3-335abcced580\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") "
Oct 30 00:32:48.890799 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.890647    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-kube-api-access-6bxjx\" (UniqueName: \"kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-kube-api-access-6bxjx\") pod \"192d192e-cab7-42b8-86d3-335abcced580\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") "
Oct 30 00:32:48.890799 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.890676    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-dns-cert\") pod \"192d192e-cab7-42b8-86d3-335abcced580\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") "
Oct 30 00:32:48.890799 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.890789    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-ca-configmap" pod "192d192e-cab7-42b8-86d3-335abcced580" (UID: "192d192e-cab7-42b8-86d3-335abcced580"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:48.891193 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.890818    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192d192e-cab7-42b8-86d3-335abcced580-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/192d192e-cab7-42b8-86d3-335abcced580-local-certs\") pod \"192d192e-cab7-42b8-86d3-335abcced580\" (UID: \"192d192e-cab7-42b8-86d3-335abcced580\") "
Oct 30 00:32:48.891642 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.891615    4441 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-ca-configmap\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:48.894964 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.894389    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-kubeconfig" pod "192d192e-cab7-42b8-86d3-335abcced580" (UID: "192d192e-cab7-42b8-86d3-335abcced580"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:48.894964 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.894746    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-cacerts" pod "192d192e-cab7-42b8-86d3-335abcced580" (UID: "192d192e-cab7-42b8-86d3-335abcced580"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:48.894964 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.894878    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192d192e-cab7-42b8-86d3-335abcced580-local-certs" pod "192d192e-cab7-42b8-86d3-335abcced580" (UID: "192d192e-cab7-42b8-86d3-335abcced580"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:48.895607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.895363    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-dns-cert" pod "192d192e-cab7-42b8-86d3-335abcced580" (UID: "192d192e-cab7-42b8-86d3-335abcced580"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:48.895607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.895434    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-kube-api-access-6bxjx" pod "192d192e-cab7-42b8-86d3-335abcced580" (UID: "192d192e-cab7-42b8-86d3-335abcced580"). InnerVolumeSpecName "kube-api-access-6bxjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:48.895607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.895501    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-istio-token" pod "192d192e-cab7-42b8-86d3-335abcced580" (UID: "192d192e-cab7-42b8-86d3-335abcced580"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:48.993655 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.993616    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:48.993655 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.993653    4441 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-kubeconfig\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:48.993886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.993669    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bxjx\" (UniqueName: \"kubernetes.io/projected/192d192e-cab7-42b8-86d3-335abcced580-kube-api-access-6bxjx\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:48.993886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.993683    4441 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-istio-csr-dns-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:48.993886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.993697    4441 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/192d192e-cab7-42b8-86d3-335abcced580-local-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:48.993886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:48.993710    4441 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/192d192e-cab7-42b8-86d3-335abcced580-cacerts\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.010286    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"]
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.010873    4441 cpu_manager.go:414] "containerMap: removing container" podUID="ab92d406-7ea3-4706-bcb6-653e403cc882" containerName="maas-api"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.010887    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab92d406-7ea3-4706-bcb6-653e403cc882" containerName="maas-api"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.010898    4441 cpu_manager.go:414] "containerMap: removing container" podUID="8030ad10-9610-4159-a607-92ca15758e63" containerName="payload-pre-processing"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.010904    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8030ad10-9610-4159-a607-92ca15758e63" containerName="payload-pre-processing"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.010928    4441 cpu_manager.go:414] "containerMap: removing container" podUID="f80a0aed-06cb-4ca3-a9e4-142a932076fe" containerName="payload-processing"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.010933    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80a0aed-06cb-4ca3-a9e4-142a932076fe" containerName="payload-processing"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.010949    4441 cpu_manager.go:414] "containerMap: removing container" podUID="42d7f0fa-9352-4408-b1a1-cc80af286710" containerName="istio-proxy"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.010954    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d7f0fa-9352-4408-b1a1-cc80af286710" containerName="istio-proxy"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.010968    4441 cpu_manager.go:414] "containerMap: removing container" podUID="98b69041-6f23-4680-a25f-2d38b2228ed3" containerName="payload-processing"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.010973    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b69041-6f23-4680-a25f-2d38b2228ed3" containerName="payload-processing"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.010986    4441 cpu_manager.go:414] "containerMap: removing container" podUID="192d192e-cab7-42b8-86d3-335abcced580" containerName="discovery"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.010991    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192d192e-cab7-42b8-86d3-335abcced580" containerName="discovery"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.011091    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b69041-6f23-4680-a25f-2d38b2228ed3" containerName="payload-processing"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.011101    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="192d192e-cab7-42b8-86d3-335abcced580" containerName="discovery"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.011111    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab92d406-7ea3-4706-bcb6-653e403cc882" containerName="maas-api"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.011117    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8030ad10-9610-4159-a607-92ca15758e63" containerName="payload-pre-processing"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.011125    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80a0aed-06cb-4ca3-a9e4-142a932076fe" containerName="payload-processing"
Oct 30 00:32:49.011481 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.011132    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d7f0fa-9352-4408-b1a1-cc80af286710" containerName="istio-proxy"
Oct 30 00:32:49.017098 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.016992    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"
Oct 30 00:32:49.050867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.047364    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-c1eeb6\""
Oct 30 00:32:49.062435 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.062107    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-c1eeb6-dockercfg-ch698\""
Oct 30 00:32:49.068157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.068067    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"]
Oct 30 00:32:49.200011 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.199969    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02017bad-fe6f-415d-a24e-4b6b26a2a99a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml\" (UID: \"02017bad-fe6f-415d-a24e-4b6b26a2a99a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"
Oct 30 00:32:49.200656 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.200629    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7kg\" (UniqueName: \"kubernetes.io/projected/02017bad-fe6f-415d-a24e-4b6b26a2a99a-kube-api-access-gv7kg\") pod \"payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml\" (UID: \"02017bad-fe6f-415d-a24e-4b6b26a2a99a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"
Oct 30 00:32:49.291207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.291124    4441 generic.go:356] "Generic (PLEG): container finished" podID="192d192e-cab7-42b8-86d3-335abcced580" containerID="eb7b71f1494879e229da0246d7e694c024ae81aa8386a3d51f273df0dddefe8c" exitCode=0
Oct 30 00:32:49.291368 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.291282    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"
Oct 30 00:32:49.291429 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.291281    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k" event={"ID":"192d192e-cab7-42b8-86d3-335abcced580","Type":"ContainerDied","Data":"eb7b71f1494879e229da0246d7e694c024ae81aa8386a3d51f273df0dddefe8c"}
Oct 30 00:32:49.291505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.291485    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k" event={"ID":"192d192e-cab7-42b8-86d3-335abcced580","Type":"ContainerDied","Data":"1dd1cd643b30c5c9bda67c90606d9d96b9baaaec7c4ce8b40ecf3b483afa87ab"}
Oct 30 00:32:49.291565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.291525    4441 scope.go:122] "RemoveContainer" containerID="eb7b71f1494879e229da0246d7e694c024ae81aa8386a3d51f273df0dddefe8c"
Oct 30 00:32:49.306238 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.302202    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02017bad-fe6f-415d-a24e-4b6b26a2a99a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml\" (UID: \"02017bad-fe6f-415d-a24e-4b6b26a2a99a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"
Oct 30 00:32:49.306238 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.302511    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7kg\" (UniqueName: \"kubernetes.io/projected/02017bad-fe6f-415d-a24e-4b6b26a2a99a-kube-api-access-gv7kg\") pod \"payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml\" (UID: \"02017bad-fe6f-415d-a24e-4b6b26a2a99a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"
Oct 30 00:32:49.306238 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.304100    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02017bad-fe6f-415d-a24e-4b6b26a2a99a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml\" (UID: \"02017bad-fe6f-415d-a24e-4b6b26a2a99a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"
Oct 30 00:32:49.321294 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.321262    4441 scope.go:122] "RemoveContainer" containerID="eb7b71f1494879e229da0246d7e694c024ae81aa8386a3d51f273df0dddefe8c"
Oct 30 00:32:49.321636 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:32:49.321615    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7b71f1494879e229da0246d7e694c024ae81aa8386a3d51f273df0dddefe8c\": container with ID starting with eb7b71f1494879e229da0246d7e694c024ae81aa8386a3d51f273df0dddefe8c not found: ID does not exist" containerID="eb7b71f1494879e229da0246d7e694c024ae81aa8386a3d51f273df0dddefe8c"
Oct 30 00:32:49.321725 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.321647    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7b71f1494879e229da0246d7e694c024ae81aa8386a3d51f273df0dddefe8c"} err="failed to get container status \"eb7b71f1494879e229da0246d7e694c024ae81aa8386a3d51f273df0dddefe8c\": rpc error: code = NotFound desc = could not find container \"eb7b71f1494879e229da0246d7e694c024ae81aa8386a3d51f273df0dddefe8c\": container with ID starting with eb7b71f1494879e229da0246d7e694c024ae81aa8386a3d51f273df0dddefe8c not found: ID does not exist"
Oct 30 00:32:49.325895 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.325868    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7kg\" (UniqueName: \"kubernetes.io/projected/02017bad-fe6f-415d-a24e-4b6b26a2a99a-kube-api-access-gv7kg\") pod \"payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml\" (UID: \"02017bad-fe6f-415d-a24e-4b6b26a2a99a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"
Oct 30 00:32:49.340800 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.340222    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"]
Oct 30 00:32:49.344115 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.344088    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bv72k"]
Oct 30 00:32:49.387822 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.387786    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"
Oct 30 00:32:49.695230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:49.695205    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"]
Oct 30 00:32:49.696333 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:32:49.696304    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02017bad_fe6f_415d_a24e_4b6b26a2a99a.slice/crio-f62bfc7303192f6401b741e8392d9b562715c213c823ef048894800c0caa5611 WatchSource:0}: Error finding container f62bfc7303192f6401b741e8392d9b562715c213c823ef048894800c0caa5611: Status 404 returned error can't find the container with id f62bfc7303192f6401b741e8392d9b562715c213c823ef048894800c0caa5611
Oct 30 00:32:50.210630 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:50.210594    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:32:50.311834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:50.311793    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml" event={"ID":"02017bad-fe6f-415d-a24e-4b6b26a2a99a","Type":"ContainerStarted","Data":"a076a8901284cf7c43af738b8ebb105160958bc58d64b8cba85a89247297c6e0"}
Oct 30 00:32:50.312122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:50.312101    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"
Oct 30 00:32:50.312265 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:50.312236    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml" event={"ID":"02017bad-fe6f-415d-a24e-4b6b26a2a99a","Type":"ContainerStarted","Data":"f62bfc7303192f6401b741e8392d9b562715c213c823ef048894800c0caa5611"}
Oct 30 00:32:50.349754 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:50.349697    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml" podStartSLOduration=2.34967929 podStartE2EDuration="2.34967929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:50.347166937 +0000 UTC m=+1679.872460844" watchObservedRunningTime="2026-10-30 00:32:50.34967929 +0000 UTC m=+1679.874973169"
Oct 30 00:32:50.611389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:50.611359    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192d192e-cab7-42b8-86d3-335abcced580" path="/var/lib/kubelet/pods/192d192e-cab7-42b8-86d3-335abcced580/volumes"
Oct 30 00:32:52.375049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:52.329089    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"]
Oct 30 00:32:52.511667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:52.511631    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"
Oct 30 00:32:52.591050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:52.580694    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"]
Oct 30 00:32:52.811396 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:52.685874    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-29f3ca-dockercfg-zn9vh\""
Oct 30 00:32:52.857048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:52.834178    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-29f3ca\""
Oct 30 00:32:52.897727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:52.887511    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:32:52.906975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:52.905841    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x\" (UID: \"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"
Oct 30 00:32:52.906975 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:52.905948    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4tzd\" (UniqueName: \"kubernetes.io/projected/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-kube-api-access-s4tzd\") pod \"payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x\" (UID: \"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"
Oct 30 00:32:52.942774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:52.941172    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:32:53.036051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.032652    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x\" (UID: \"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"
Oct 30 00:32:53.051047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.036414    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4tzd\" (UniqueName: \"kubernetes.io/projected/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-kube-api-access-s4tzd\") pod \"payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x\" (UID: \"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"
Oct 30 00:32:53.051344 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.043728    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x\" (UID: \"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"
Oct 30 00:32:53.206641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.200536    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:32:53.327046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.309066    4441 scope.go:122] "RemoveContainer" containerID="a4066e887ee99a3e82988b505af53db4b0797d2632301bc2b61603c30a991f5d"
Oct 30 00:32:53.327046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.310871    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 00:32:53.327046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.310872    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:32:53.340072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.339759    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"]
Oct 30 00:32:53.340072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.339996    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t" podUID="98b0e606-d290-40ee-a057-77633a3e07bf" containerName="istio-proxy" containerID="cri-o://f1c7d75d477924d88acd073eaa280b0d43d5402ccff2db13891e26754317a6e8" gracePeriod=30
Oct 30 00:32:53.469449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.469344    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:32:53.470393 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.470350    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:32:53.470907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.470882    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:32:53.471423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.471401    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:32:53.512793 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.512728    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4tzd\" (UniqueName: \"kubernetes.io/projected/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-kube-api-access-s4tzd\") pod \"payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x\" (UID: \"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"
Oct 30 00:32:53.735391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.735358    4441 scope.go:122] "RemoveContainer" containerID="46cfc9cb1947101b34936358cc57f92015e6560d69246247dd9f32de9fa14e3d"
Oct 30 00:32:53.736512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.736488    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"
Oct 30 00:32:53.779057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.776313    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:32:53.892243 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.892200    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:32:53.920050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.893051    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:32:53.942195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.942157    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 00:32:53.953180 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:53.943070    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:32:54.241787 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.178642    4441 generic.go:356] "Generic (PLEG): container finished" podID="4489355f-aad2-4394-bd10-7a78024d6458" containerID="9ec72f9547d83ad4ee92e17fd4e5e961e148b98f9ac2cafc80a6202423eecd11" exitCode=0
Oct 30 00:32:54.255558 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.207518    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:32:54.261907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.252748    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw" event={"ID":"4489355f-aad2-4394-bd10-7a78024d6458","Type":"ContainerDied","Data":"9ec72f9547d83ad4ee92e17fd4e5e961e148b98f9ac2cafc80a6202423eecd11"}
Oct 30 00:32:54.262493 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.262436    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:32:54.405155 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.405114    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:32:54.526863 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.525139    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:32:54.526863 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.525425    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:32:54.526863 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.526010    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:32:54.526863 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.526085    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:32:54.557589 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.538563    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:32:54.557589 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.553663    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:32:54.597581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.597539    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:32:54.671422 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.671391    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:32:54.671591 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.671451    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:32:54.700418 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.700303    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"]
Oct 30 00:32:54.707337 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.705356    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:32:54.731328 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.731116    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-socket\") pod \"4489355f-aad2-4394-bd10-7a78024d6458\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") "
Oct 30 00:32:54.741610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.735411    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-envoy\") pod \"4489355f-aad2-4394-bd10-7a78024d6458\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") "
Oct 30 00:32:54.741610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.735451    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4489355f-aad2-4394-bd10-7a78024d6458-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4489355f-aad2-4394-bd10-7a78024d6458-istiod-ca-cert\") pod \"4489355f-aad2-4394-bd10-7a78024d6458\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") "
Oct 30 00:32:54.741610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.735536    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-istio-token\" (UniqueName: \"kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-istio-token\") pod \"4489355f-aad2-4394-bd10-7a78024d6458\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") "
Oct 30 00:32:54.741610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.735617    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-kube-api-access-4lb2r\" (UniqueName: \"kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-kube-api-access-4lb2r\") pod \"4489355f-aad2-4394-bd10-7a78024d6458\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") "
Oct 30 00:32:54.741610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.735770    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4489355f-aad2-4394-bd10-7a78024d6458-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4489355f-aad2-4394-bd10-7a78024d6458-istio-podinfo\") pod \"4489355f-aad2-4394-bd10-7a78024d6458\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") "
Oct 30 00:32:54.741610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.735819    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-certs\") pod \"4489355f-aad2-4394-bd10-7a78024d6458\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") "
Oct 30 00:32:54.741610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.735858    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-credential-socket\") pod \"4489355f-aad2-4394-bd10-7a78024d6458\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") "
Oct 30 00:32:54.741610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.735945    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-data\") pod \"4489355f-aad2-4394-bd10-7a78024d6458\" (UID: \"4489355f-aad2-4394-bd10-7a78024d6458\") "
Oct 30 00:32:54.741610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.735318    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-socket" pod "4489355f-aad2-4394-bd10-7a78024d6458" (UID: "4489355f-aad2-4394-bd10-7a78024d6458"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:54.741610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.738368    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4489355f-aad2-4394-bd10-7a78024d6458-istiod-ca-cert" pod "4489355f-aad2-4394-bd10-7a78024d6458" (UID: "4489355f-aad2-4394-bd10-7a78024d6458"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:54.741610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.738600    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-certs" pod "4489355f-aad2-4394-bd10-7a78024d6458" (UID: "4489355f-aad2-4394-bd10-7a78024d6458"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:54.741610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.740330    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-credential-socket" pod "4489355f-aad2-4394-bd10-7a78024d6458" (UID: "4489355f-aad2-4394-bd10-7a78024d6458"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:54.741610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.741090    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-data" pod "4489355f-aad2-4394-bd10-7a78024d6458" (UID: "4489355f-aad2-4394-bd10-7a78024d6458"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:54.748553 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.746304    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-istio-token" pod "4489355f-aad2-4394-bd10-7a78024d6458" (UID: "4489355f-aad2-4394-bd10-7a78024d6458"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:54.749203 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.749092    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4489355f-aad2-4394-bd10-7a78024d6458-istio-podinfo" pod "4489355f-aad2-4394-bd10-7a78024d6458" (UID: "4489355f-aad2-4394-bd10-7a78024d6458"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:32:54.750178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.750035    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-kube-api-access-4lb2r" pod "4489355f-aad2-4394-bd10-7a78024d6458" (UID: "4489355f-aad2-4394-bd10-7a78024d6458"). InnerVolumeSpecName "kube-api-access-4lb2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:54.750950 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.750916    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-envoy" pod "4489355f-aad2-4394-bd10-7a78024d6458" (UID: "4489355f-aad2-4394-bd10-7a78024d6458"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:54.843658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.843629    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.843658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.843659    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.843844 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.843672    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.843844 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.843685    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.843844 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.843696    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4489355f-aad2-4394-bd10-7a78024d6458-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.843844 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.843708    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4489355f-aad2-4394-bd10-7a78024d6458-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.843844 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.843719    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.843844 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.843732    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lb2r\" (UniqueName: \"kubernetes.io/projected/4489355f-aad2-4394-bd10-7a78024d6458-kube-api-access-4lb2r\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.843844 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:54.843744    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4489355f-aad2-4394-bd10-7a78024d6458-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:55.279724 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:55.279679    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x" event={"ID":"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c","Type":"ContainerStarted","Data":"70fe7ce9f2e3e02325646ec405c8d7346bf768a63c4ecbf26cf5a11ac7a5a8ef"}
Oct 30 00:32:55.280101 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:55.279750    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x" event={"ID":"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c","Type":"ContainerStarted","Data":"c2caf619da3b7e4720560d185faa9efce1dd96032e9fdd4e166dc91209049f5b"}
Oct 30 00:32:55.280194 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:55.280151    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"
Oct 30 00:32:55.287302 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:55.287147    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"
Oct 30 00:32:55.287947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:55.287926    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw" event={"ID":"4489355f-aad2-4394-bd10-7a78024d6458","Type":"ContainerDied","Data":"872dd15b51cace86314508034b4981efe7a6f512881c213f0dc01b07b8b5ca61"}
Oct 30 00:32:55.287996 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:55.287972    4441 scope.go:122] "RemoveContainer" containerID="9ec72f9547d83ad4ee92e17fd4e5e961e148b98f9ac2cafc80a6202423eecd11"
Oct 30 00:32:55.323015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:55.321735    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x" podStartSLOduration=3.321716815 podStartE2EDuration="3.321716815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:55.316879933 +0000 UTC m=+1684.842173813" watchObservedRunningTime="2026-10-30 00:32:55.321716815 +0000 UTC m=+1684.847010692"
Oct 30 00:32:55.352803 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:55.352768    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"]
Oct 30 00:32:55.357986 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:55.357957    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jhwdw"]
Oct 30 00:32:56.610473 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:56.610444    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4489355f-aad2-4394-bd10-7a78024d6458" path="/var/lib/kubelet/pods/4489355f-aad2-4394-bd10-7a78024d6458/volumes"
Oct 30 00:32:58.762954 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.762857    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:32:58.805214 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.805116    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-istio-token\" (UniqueName: \"kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-istio-token\") pod \"98b0e606-d290-40ee-a057-77633a3e07bf\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") "
Oct 30 00:32:58.805431 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.805305    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-credential-socket\") pod \"98b0e606-d290-40ee-a057-77633a3e07bf\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") "
Oct 30 00:32:58.805431 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.805341    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-socket\") pod \"98b0e606-d290-40ee-a057-77633a3e07bf\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") "
Oct 30 00:32:58.805431 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.805410    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98b0e606-d290-40ee-a057-77633a3e07bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98b0e606-d290-40ee-a057-77633a3e07bf-istio-podinfo\") pod \"98b0e606-d290-40ee-a057-77633a3e07bf\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") "
Oct 30 00:32:58.805651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.805488    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98b0e606-d290-40ee-a057-77633a3e07bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98b0e606-d290-40ee-a057-77633a3e07bf-istiod-ca-cert\") pod \"98b0e606-d290-40ee-a057-77633a3e07bf\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") "
Oct 30 00:32:58.805651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.805520    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-data\") pod \"98b0e606-d290-40ee-a057-77633a3e07bf\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") "
Oct 30 00:32:58.805774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.805724    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-socket" pod "98b0e606-d290-40ee-a057-77633a3e07bf" (UID: "98b0e606-d290-40ee-a057-77633a3e07bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:58.805904 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.805878    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-credential-socket" pod "98b0e606-d290-40ee-a057-77633a3e07bf" (UID: "98b0e606-d290-40ee-a057-77633a3e07bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:58.806201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.806167    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-certs\") pod \"98b0e606-d290-40ee-a057-77633a3e07bf\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") "
Oct 30 00:32:58.806340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.806269    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-data" pod "98b0e606-d290-40ee-a057-77633a3e07bf" (UID: "98b0e606-d290-40ee-a057-77633a3e07bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:58.806340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.806299    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-kube-api-access-2frxn\" (UniqueName: \"kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-kube-api-access-2frxn\") pod \"98b0e606-d290-40ee-a057-77633a3e07bf\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") "
Oct 30 00:32:58.806340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.806304    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b0e606-d290-40ee-a057-77633a3e07bf-istiod-ca-cert" pod "98b0e606-d290-40ee-a057-77633a3e07bf" (UID: "98b0e606-d290-40ee-a057-77633a3e07bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:58.806340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.806338    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-envoy\") pod \"98b0e606-d290-40ee-a057-77633a3e07bf\" (UID: \"98b0e606-d290-40ee-a057-77633a3e07bf\") "
Oct 30 00:32:58.806881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.806859    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:58.806982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.806883    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98b0e606-d290-40ee-a057-77633a3e07bf-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:58.806982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.806897    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:58.808038 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.807983    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-certs" pod "98b0e606-d290-40ee-a057-77633a3e07bf" (UID: "98b0e606-d290-40ee-a057-77633a3e07bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:58.809178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.809155    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98b0e606-d290-40ee-a057-77633a3e07bf-istio-podinfo" pod "98b0e606-d290-40ee-a057-77633a3e07bf" (UID: "98b0e606-d290-40ee-a057-77633a3e07bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:32:58.809496 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.809472    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-istio-token" pod "98b0e606-d290-40ee-a057-77633a3e07bf" (UID: "98b0e606-d290-40ee-a057-77633a3e07bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:58.812479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.812177    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-envoy" pod "98b0e606-d290-40ee-a057-77633a3e07bf" (UID: "98b0e606-d290-40ee-a057-77633a3e07bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:58.812479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.812278    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-kube-api-access-2frxn" pod "98b0e606-d290-40ee-a057-77633a3e07bf" (UID: "98b0e606-d290-40ee-a057-77633a3e07bf"). InnerVolumeSpecName "kube-api-access-2frxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:58.908012 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.907980    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2frxn\" (UniqueName: \"kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-kube-api-access-2frxn\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:58.908012 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.908009    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:58.908216 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.908019    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98b0e606-d290-40ee-a057-77633a3e07bf-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:58.908216 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.908043    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98b0e606-d290-40ee-a057-77633a3e07bf-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:58.908216 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.908051    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:58.908216 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:58.908059    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98b0e606-d290-40ee-a057-77633a3e07bf-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:32:59.322251 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:59.322210    4441 generic.go:356] "Generic (PLEG): container finished" podID="98b0e606-d290-40ee-a057-77633a3e07bf" containerID="f1c7d75d477924d88acd073eaa280b0d43d5402ccff2db13891e26754317a6e8" exitCode=0
Oct 30 00:32:59.322449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:59.322256    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"
Oct 30 00:32:59.322449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:59.322287    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t" event={"ID":"98b0e606-d290-40ee-a057-77633a3e07bf","Type":"ContainerDied","Data":"f1c7d75d477924d88acd073eaa280b0d43d5402ccff2db13891e26754317a6e8"}
Oct 30 00:32:59.322449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:59.322352    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t" event={"ID":"98b0e606-d290-40ee-a057-77633a3e07bf","Type":"ContainerDied","Data":"8343d3d3167ea9bf29f1f44bbcb4fac62d6aa8e26ce502011f184c81f37def5a"}
Oct 30 00:32:59.322449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:59.322383    4441 scope.go:122] "RemoveContainer" containerID="f1c7d75d477924d88acd073eaa280b0d43d5402ccff2db13891e26754317a6e8"
Oct 30 00:32:59.335303 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:59.335279    4441 scope.go:122] "RemoveContainer" containerID="f1c7d75d477924d88acd073eaa280b0d43d5402ccff2db13891e26754317a6e8"
Oct 30 00:32:59.335635 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:32:59.335607    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c7d75d477924d88acd073eaa280b0d43d5402ccff2db13891e26754317a6e8\": container with ID starting with f1c7d75d477924d88acd073eaa280b0d43d5402ccff2db13891e26754317a6e8 not found: ID does not exist" containerID="f1c7d75d477924d88acd073eaa280b0d43d5402ccff2db13891e26754317a6e8"
Oct 30 00:32:59.335733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:59.335644    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c7d75d477924d88acd073eaa280b0d43d5402ccff2db13891e26754317a6e8"} err="failed to get container status \"f1c7d75d477924d88acd073eaa280b0d43d5402ccff2db13891e26754317a6e8\": rpc error: code = NotFound desc = could not find container \"f1c7d75d477924d88acd073eaa280b0d43d5402ccff2db13891e26754317a6e8\": container with ID starting with f1c7d75d477924d88acd073eaa280b0d43d5402ccff2db13891e26754317a6e8 not found: ID does not exist"
Oct 30 00:32:59.371911 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:59.371889    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"]
Oct 30 00:32:59.383000 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:32:59.382963    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n8j6t"]
Oct 30 00:33:00.610384 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:00.610352    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98b0e606-d290-40ee-a057-77633a3e07bf" path="/var/lib/kubelet/pods/98b0e606-d290-40ee-a057-77633a3e07bf/volumes"
Oct 30 00:33:01.349331 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.349302    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"
Oct 30 00:33:01.645673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.645593    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"]
Oct 30 00:33:01.646405 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.646385    4441 cpu_manager.go:414] "containerMap: removing container" podUID="4489355f-aad2-4394-bd10-7a78024d6458" containerName="istio-proxy"
Oct 30 00:33:01.646405 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.646405    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4489355f-aad2-4394-bd10-7a78024d6458" containerName="istio-proxy"
Oct 30 00:33:01.646572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.646461    4441 cpu_manager.go:414] "containerMap: removing container" podUID="98b0e606-d290-40ee-a057-77633a3e07bf" containerName="istio-proxy"
Oct 30 00:33:01.646572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.646469    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b0e606-d290-40ee-a057-77633a3e07bf" containerName="istio-proxy"
Oct 30 00:33:01.646673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.646616    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b0e606-d290-40ee-a057-77633a3e07bf" containerName="istio-proxy"
Oct 30 00:33:01.646673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.646639    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4489355f-aad2-4394-bd10-7a78024d6458" containerName="istio-proxy"
Oct 30 00:33:01.651545 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.651522    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"
Oct 30 00:33:01.665508 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.665489    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-63e288\""
Oct 30 00:33:01.671816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.670006    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"]
Oct 30 00:33:01.717608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.717578    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"]
Oct 30 00:33:01.721697 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.721681    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"
Oct 30 00:33:01.740510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.740490    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"]
Oct 30 00:33:01.763093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.758009    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187932ee-ad15-4170-9e9f-13dc02fa8756-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872\" (UID: \"187932ee-ad15-4170-9e9f-13dc02fa8756\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"
Oct 30 00:33:01.763093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.758248    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp6l2\" (UniqueName: \"kubernetes.io/projected/187932ee-ad15-4170-9e9f-13dc02fa8756-kube-api-access-kp6l2\") pod \"payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872\" (UID: \"187932ee-ad15-4170-9e9f-13dc02fa8756\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"
Oct 30 00:33:01.793452 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.793418    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-63e288-dockercfg-hgfdl\""
Oct 30 00:33:01.880075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.875848    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bzqb\" (UniqueName: \"kubernetes.io/projected/6df19eba-a75b-49a8-9c52-fd7000d43e75-kube-api-access-9bzqb\") pod \"payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg\" (UID: \"6df19eba-a75b-49a8-9c52-fd7000d43e75\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"
Oct 30 00:33:01.880075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.875985    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187932ee-ad15-4170-9e9f-13dc02fa8756-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872\" (UID: \"187932ee-ad15-4170-9e9f-13dc02fa8756\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"
Oct 30 00:33:01.880075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.876136    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp6l2\" (UniqueName: \"kubernetes.io/projected/187932ee-ad15-4170-9e9f-13dc02fa8756-kube-api-access-kp6l2\") pod \"payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872\" (UID: \"187932ee-ad15-4170-9e9f-13dc02fa8756\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"
Oct 30 00:33:01.880075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.876174    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df19eba-a75b-49a8-9c52-fd7000d43e75-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg\" (UID: \"6df19eba-a75b-49a8-9c52-fd7000d43e75\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"
Oct 30 00:33:01.880075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.877170    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187932ee-ad15-4170-9e9f-13dc02fa8756-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872\" (UID: \"187932ee-ad15-4170-9e9f-13dc02fa8756\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"
Oct 30 00:33:01.910092 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.903201    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp6l2\" (UniqueName: \"kubernetes.io/projected/187932ee-ad15-4170-9e9f-13dc02fa8756-kube-api-access-kp6l2\") pod \"payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872\" (UID: \"187932ee-ad15-4170-9e9f-13dc02fa8756\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"
Oct 30 00:33:01.970477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.970440    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"
Oct 30 00:33:01.977880 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.977846    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bzqb\" (UniqueName: \"kubernetes.io/projected/6df19eba-a75b-49a8-9c52-fd7000d43e75-kube-api-access-9bzqb\") pod \"payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg\" (UID: \"6df19eba-a75b-49a8-9c52-fd7000d43e75\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"
Oct 30 00:33:01.978126 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.978108    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df19eba-a75b-49a8-9c52-fd7000d43e75-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg\" (UID: \"6df19eba-a75b-49a8-9c52-fd7000d43e75\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"
Oct 30 00:33:01.978796 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:01.978774    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df19eba-a75b-49a8-9c52-fd7000d43e75-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg\" (UID: \"6df19eba-a75b-49a8-9c52-fd7000d43e75\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"
Oct 30 00:33:02.002490 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.002452    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bzqb\" (UniqueName: \"kubernetes.io/projected/6df19eba-a75b-49a8-9c52-fd7000d43e75-kube-api-access-9bzqb\") pod \"payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg\" (UID: \"6df19eba-a75b-49a8-9c52-fd7000d43e75\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"
Oct 30 00:33:02.035942 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.035900    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"
Oct 30 00:33:02.155259 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.155225    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"]
Oct 30 00:33:02.160591 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.160537    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.174121 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.174079    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-c1eeb6-serving-cert\""
Oct 30 00:33:02.177692 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.177387    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"]
Oct 30 00:33:02.283237 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.283208    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.283434 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.283273    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pps9m\" (UniqueName: \"kubernetes.io/projected/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-kube-api-access-pps9m\") pod \"maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.283434 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.283410    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.283536 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.283449    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.292482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.292448    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"]
Oct 30 00:33:02.293578 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:02.293553    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187932ee_ad15_4170_9e9f_13dc02fa8756.slice/crio-9d96bedbf43d02e63ca3d2c1e7b0e7196dae5880ca1a04df1420a8b0b64bdabb WatchSource:0}: Error finding container 9d96bedbf43d02e63ca3d2c1e7b0e7196dae5880ca1a04df1420a8b0b64bdabb: Status 404 returned error can't find the container with id 9d96bedbf43d02e63ca3d2c1e7b0e7196dae5880ca1a04df1420a8b0b64bdabb
Oct 30 00:33:02.352807 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.352767    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872" event={"ID":"187932ee-ad15-4170-9e9f-13dc02fa8756","Type":"ContainerStarted","Data":"9d96bedbf43d02e63ca3d2c1e7b0e7196dae5880ca1a04df1420a8b0b64bdabb"}
Oct 30 00:33:02.377765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.377732    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"]
Oct 30 00:33:02.378985 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:02.378949    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df19eba_a75b_49a8_9c52_fd7000d43e75.slice/crio-c7585a6989e3da733b8798c1169f830b3970bc0f4f606c80d82ed19f2c38080b WatchSource:0}: Error finding container c7585a6989e3da733b8798c1169f830b3970bc0f4f606c80d82ed19f2c38080b: Status 404 returned error can't find the container with id c7585a6989e3da733b8798c1169f830b3970bc0f4f606c80d82ed19f2c38080b
Oct 30 00:33:02.387071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.384579    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.387071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.384672    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pps9m\" (UniqueName: \"kubernetes.io/projected/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-kube-api-access-pps9m\") pod \"maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.387071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.384755    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.387071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.384783    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.387071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.385823    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.397177 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.390147    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.401390 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.401331    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.421960 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.421883    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pps9m\" (UniqueName: \"kubernetes.io/projected/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-kube-api-access-pps9m\") pod \"maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.493058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.493019    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:02.795465 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:02.795436    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"]
Oct 30 00:33:02.796135 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:02.796113    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0d0e19_9703_41aa_8a3c_4e0a85057b94.slice/crio-b000c9288d088aa12576e41d0ceec3ee07903cab44b1078d555b75e22dd230f0 WatchSource:0}: Error finding container b000c9288d088aa12576e41d0ceec3ee07903cab44b1078d555b75e22dd230f0: Status 404 returned error can't find the container with id b000c9288d088aa12576e41d0ceec3ee07903cab44b1078d555b75e22dd230f0
Oct 30 00:33:03.362073 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:03.361999    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz" event={"ID":"bc0d0e19-9703-41aa-8a3c-4e0a85057b94","Type":"ContainerStarted","Data":"27cf173a0cd31d6dcac7c491782c4411d4881c2629fa6e69e1cecd0c583a6442"}
Oct 30 00:33:03.362257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:03.362108    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz" event={"ID":"bc0d0e19-9703-41aa-8a3c-4e0a85057b94","Type":"ContainerStarted","Data":"b000c9288d088aa12576e41d0ceec3ee07903cab44b1078d555b75e22dd230f0"}
Oct 30 00:33:03.362257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:03.362145    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:03.364491 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:03.364461    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872" event={"ID":"187932ee-ad15-4170-9e9f-13dc02fa8756","Type":"ContainerStarted","Data":"c8e1b4083554026cd0d7626531434c00e80ecaa08a0ac3a220409e960cf30a59"}
Oct 30 00:33:03.364656 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:03.364512    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"
Oct 30 00:33:03.366209 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:03.366178    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg" event={"ID":"6df19eba-a75b-49a8-9c52-fd7000d43e75","Type":"ContainerStarted","Data":"d7899ddfce98043e050200e85ec196631fd0110d46fbbb21e4becb4fcc08b1ee"}
Oct 30 00:33:03.366310 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:03.366229    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg" event={"ID":"6df19eba-a75b-49a8-9c52-fd7000d43e75","Type":"ContainerStarted","Data":"c7585a6989e3da733b8798c1169f830b3970bc0f4f606c80d82ed19f2c38080b"}
Oct 30 00:33:03.366310 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:03.366257    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"
Oct 30 00:33:03.405991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:03.405926    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz" podStartSLOduration=0.986957399 podStartE2EDuration="1.405909185s" totalImagesPullingTime="418.951786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:03.399713715 +0000 UTC m=+1692.925007614" watchObservedRunningTime="2026-10-30 00:33:03.405909185 +0000 UTC m=+1692.931203063"
Oct 30 00:33:03.444364 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:03.444281    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg" podStartSLOduration=2.444259116 podStartE2EDuration="2.444259116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:03.436539986 +0000 UTC m=+1692.961833866" watchObservedRunningTime="2026-10-30 00:33:03.444259116 +0000 UTC m=+1692.969552994"
Oct 30 00:33:03.472475 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:03.472358    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872" podStartSLOduration=2.472341177 podStartE2EDuration="2.472341177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:03.471934095 +0000 UTC m=+1692.997227973" watchObservedRunningTime="2026-10-30 00:33:03.472341177 +0000 UTC m=+1692.997635055"
Oct 30 00:33:03.903840 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:33:03.903797    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:33:05.042850 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.042816    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"]
Oct 30 00:33:05.050317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.050294    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.091327 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.091283    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-29f3ca-serving-cert\""
Oct 30 00:33:05.097950 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.097918    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"]
Oct 30 00:33:05.124623 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.124590    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8htpg\" (UniqueName: \"kubernetes.io/projected/4e076602-4664-4841-8894-75908287152d-kube-api-access-8htpg\") pod \"maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2\" (UID: \"4e076602-4664-4841-8894-75908287152d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.124803 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.124731    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e076602-4664-4841-8894-75908287152d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2\" (UID: \"4e076602-4664-4841-8894-75908287152d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.124803 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.124755    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2\" (UID: \"4e076602-4664-4841-8894-75908287152d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.124803 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.124785    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2\" (UID: \"4e076602-4664-4841-8894-75908287152d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.227049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.226359    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8htpg\" (UniqueName: \"kubernetes.io/projected/4e076602-4664-4841-8894-75908287152d-kube-api-access-8htpg\") pod \"maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2\" (UID: \"4e076602-4664-4841-8894-75908287152d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.227049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.226515    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e076602-4664-4841-8894-75908287152d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2\" (UID: \"4e076602-4664-4841-8894-75908287152d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.227049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.226550    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2\" (UID: \"4e076602-4664-4841-8894-75908287152d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.227049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.226594    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2\" (UID: \"4e076602-4664-4841-8894-75908287152d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.227616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.227584    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e076602-4664-4841-8894-75908287152d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2\" (UID: \"4e076602-4664-4841-8894-75908287152d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.244130 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.244099    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2\" (UID: \"4e076602-4664-4841-8894-75908287152d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.246540 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.246515    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2\" (UID: \"4e076602-4664-4841-8894-75908287152d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.248850 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.248830    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8htpg\" (UniqueName: \"kubernetes.io/projected/4e076602-4664-4841-8894-75908287152d-kube-api-access-8htpg\") pod \"maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2\" (UID: \"4e076602-4664-4841-8894-75908287152d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.367134 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.367052    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:05.674838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:05.674810    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"]
Oct 30 00:33:05.677331 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:05.677299    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e076602_4664_4841_8894_75908287152d.slice/crio-dedcff339cb4621266a438bed7cd7ac0439ab6dd0c1f13fdfe1f6f73cd208fed WatchSource:0}: Error finding container dedcff339cb4621266a438bed7cd7ac0439ab6dd0c1f13fdfe1f6f73cd208fed: Status 404 returned error can't find the container with id dedcff339cb4621266a438bed7cd7ac0439ab6dd0c1f13fdfe1f6f73cd208fed
Oct 30 00:33:06.299048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:06.298947    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"
Oct 30 00:33:06.391286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:06.391253    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2" event={"ID":"4e076602-4664-4841-8894-75908287152d","Type":"ContainerStarted","Data":"a734ad563fc54178dc077791a19a834f10d12838d54969931da82201f771d0e8"}
Oct 30 00:33:06.391456 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:06.391309    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2" event={"ID":"4e076602-4664-4841-8894-75908287152d","Type":"ContainerStarted","Data":"dedcff339cb4621266a438bed7cd7ac0439ab6dd0c1f13fdfe1f6f73cd208fed"}
Oct 30 00:33:06.391544 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:06.391487    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:06.428590 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:06.428522    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2" podStartSLOduration=1.096032799 podStartE2EDuration="1.428508821s" totalImagesPullingTime="332.476022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:06.428318095 +0000 UTC m=+1695.953611978" watchObservedRunningTime="2026-10-30 00:33:06.428508821 +0000 UTC m=+1695.953802698"
Oct 30 00:33:09.379848 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.379816    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:33:09.520902 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.520867    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"]
Oct 30 00:33:09.525539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.525519    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:09.572496 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.572198    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-a782e5-serving-cert\""
Oct 30 00:33:09.575547 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.575516    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"]
Oct 30 00:33:09.694704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.694631    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:09.694704 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.694675    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:09.694927 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.694897    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:09.695168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.695150    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkm7j\" (UniqueName: \"kubernetes.io/projected/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-kube-api-access-jkm7j\") pod \"maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:09.800360 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.800321    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:09.800896 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.800872    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:09.801098 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.801067    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkm7j\" (UniqueName: \"kubernetes.io/projected/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-kube-api-access-jkm7j\") pod \"maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:09.801213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.801155    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:09.803351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.803006    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:09.806165 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.806118    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:09.807825 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.807787    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:09.826001 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.825970    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkm7j\" (UniqueName: \"kubernetes.io/projected/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-kube-api-access-jkm7j\") pod \"maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:09.872482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:09.872446    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:10.193594 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:10.192351    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"]
Oct 30 00:33:10.193594 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:10.192756    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7ee5a9_ec28_41f7_8dcf_0a88595c3400.slice/crio-85e32eacdfbd84480d4af81f0770ded7151ea3cd53782db6bd14dbedffda7a17 WatchSource:0}: Error finding container 85e32eacdfbd84480d4af81f0770ded7151ea3cd53782db6bd14dbedffda7a17: Status 404 returned error can't find the container with id 85e32eacdfbd84480d4af81f0770ded7151ea3cd53782db6bd14dbedffda7a17
Oct 30 00:33:10.422955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:10.422914    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg" event={"ID":"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400","Type":"ContainerStarted","Data":"85e32eacdfbd84480d4af81f0770ded7151ea3cd53782db6bd14dbedffda7a17"}
Oct 30 00:33:11.433207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:11.433167    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg" event={"ID":"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400","Type":"ContainerStarted","Data":"b6fabaf38b859ac8a0c951f6154619997531c23c7808fe966a925ffecd42beb8"}
Oct 30 00:33:11.433691 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:11.433262    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:11.470796 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:11.470750    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg" podStartSLOduration=2.095566621 podStartE2EDuration="2.47073842s" totalImagesPullingTime="375.171799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:11.470156959 +0000 UTC m=+1700.995450849" watchObservedRunningTime="2026-10-30 00:33:11.47073842 +0000 UTC m=+1700.996032297"
Oct 30 00:33:12.405389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:12.405363    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:33:14.377554 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:14.377525    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"
Oct 30 00:33:14.377992 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:14.377597    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"
Oct 30 00:33:15.475414 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.475384    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"]
Oct 30 00:33:15.479572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.479553    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:15.490476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.490169    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwstf\" (UniqueName: \"kubernetes.io/projected/1d5c887e-c6d9-41b0-8b98-78428e123af5-kube-api-access-vwstf\") pod \"maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:15.490476 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.490245    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5c887e-c6d9-41b0-8b98-78428e123af5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:15.490623 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.490526    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:15.490719 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.490701    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:15.510071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.503995    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-63e288-serving-cert\""
Oct 30 00:33:15.510071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.507905    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"]
Oct 30 00:33:15.592306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.592273    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:15.592458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.592402    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:15.592534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.592516    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwstf\" (UniqueName: \"kubernetes.io/projected/1d5c887e-c6d9-41b0-8b98-78428e123af5-kube-api-access-vwstf\") pod \"maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:15.592617 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.592601    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5c887e-c6d9-41b0-8b98-78428e123af5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:15.593331 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.593304    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5c887e-c6d9-41b0-8b98-78428e123af5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:15.594771 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.594747    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:15.594771 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.594765    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:15.613108 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.613089    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwstf\" (UniqueName: \"kubernetes.io/projected/1d5c887e-c6d9-41b0-8b98-78428e123af5-kube-api-access-vwstf\") pod \"maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:15.798420 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:15.798383    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:16.111681 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.111657    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"]
Oct 30 00:33:16.113353 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:16.113317    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5c887e_c6d9_41b0_8b98_78428e123af5.slice/crio-da10f4363e0ad05bdcf8192b29d261b8993a9e0abb2852467d24a817db099f78 WatchSource:0}: Error finding container da10f4363e0ad05bdcf8192b29d261b8993a9e0abb2852467d24a817db099f78: Status 404 returned error can't find the container with id da10f4363e0ad05bdcf8192b29d261b8993a9e0abb2852467d24a817db099f78
Oct 30 00:33:16.482472 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.482420    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz" event={"ID":"1d5c887e-c6d9-41b0-8b98-78428e123af5","Type":"ContainerStarted","Data":"da10f4363e0ad05bdcf8192b29d261b8993a9e0abb2852467d24a817db099f78"}
Oct 30 00:33:16.612530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.612497    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"]
Oct 30 00:33:16.617389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.617368    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.649337 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.642554    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"]
Oct 30 00:33:16.708711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.708623    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxxjn\" (UniqueName: \"kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-kube-api-access-zxxjn\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.708711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.708684    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.708711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.708709    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.709057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.708753    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.709057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.708797    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.709057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.708856    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.709057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.708876    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.709057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.708947    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.709057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.708972    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.810717 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.810672    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.810717 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.810723    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.810984 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.810781    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.810984 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.810839    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.810984 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.810924    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.810984 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.810948    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.811297 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.811068    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.811297 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.811106    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.811430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.811289    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.811430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.811260    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxjn\" (UniqueName: \"kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-kube-api-access-zxxjn\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.811539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.811502    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.811717 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.811689    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.811807 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.811738    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.812323 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.812301    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.814160 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.814133    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.814263 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.814245    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.832156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.832129    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.832389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.832356    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxjn\" (UniqueName: \"kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-kube-api-access-zxxjn\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:16.938512 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:16.937118    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:17.281430 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:17.281388    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"]
Oct 30 00:33:17.281569 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:17.281547    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2b1b2b_cc9e_4159_a4ab_247692d92831.slice/crio-5bcb9296c6947500418b8195ce6f1ff3554aa5dda82d2705fa84a3fab8f76d8f WatchSource:0}: Error finding container 5bcb9296c6947500418b8195ce6f1ff3554aa5dda82d2705fa84a3fab8f76d8f: Status 404 returned error can't find the container with id 5bcb9296c6947500418b8195ce6f1ff3554aa5dda82d2705fa84a3fab8f76d8f
Oct 30 00:33:17.284347 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:17.284313    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:17.284459 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:17.284385    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:17.284459 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:17.284418    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:17.472673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:17.472642    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:33:17.494284 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:17.494246    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" event={"ID":"ca2b1b2b-cc9e-4159-a4ab-247692d92831","Type":"ContainerStarted","Data":"377977b3643a7c53a2eef8aded0b1afcb85d65c601dc20dc0b6139c5cc0814c9"}
Oct 30 00:33:17.494939 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:17.494314    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" event={"ID":"ca2b1b2b-cc9e-4159-a4ab-247692d92831","Type":"ContainerStarted","Data":"5bcb9296c6947500418b8195ce6f1ff3554aa5dda82d2705fa84a3fab8f76d8f"}
Oct 30 00:33:17.503186 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:17.497770    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz" event={"ID":"1d5c887e-c6d9-41b0-8b98-78428e123af5","Type":"ContainerStarted","Data":"73adc4398cc61c562ed4925e87f5cd9bd60c2ce8c48abc822d90a75f6f05ff86"}
Oct 30 00:33:17.503186 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:17.498604    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:17.617831 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:17.617720    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz" podStartSLOduration=2.263478272 podStartE2EDuration="2.617700601s" totalImagesPullingTime="354.222329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:17.556398531 +0000 UTC m=+1707.081692408" watchObservedRunningTime="2026-10-30 00:33:17.617700601 +0000 UTC m=+1707.142994481"
Oct 30 00:33:17.622889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:17.622815    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" podStartSLOduration=1.622803463 podStartE2EDuration="1.622803463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:17.619354809 +0000 UTC m=+1707.144648687" watchObservedRunningTime="2026-10-30 00:33:17.622803463 +0000 UTC m=+1707.148097343"
Oct 30 00:33:17.938071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:17.937951    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:18.944078 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:18.943673    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 00:33:18.944078 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:18.943775    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" podUID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 00:33:19.572130 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.572091    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"]
Oct 30 00:33:19.589073 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.588873    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"]
Oct 30 00:33:19.589073 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.589000    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"
Oct 30 00:33:19.605441 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.601254    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg48v\" (UniqueName: \"kubernetes.io/projected/1910466b-ea6b-4193-8242-e117d64a1f3e-kube-api-access-kg48v\") pod \"payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5\" (UID: \"1910466b-ea6b-4193-8242-e117d64a1f3e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"
Oct 30 00:33:19.605441 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.601475    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1910466b-ea6b-4193-8242-e117d64a1f3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5\" (UID: \"1910466b-ea6b-4193-8242-e117d64a1f3e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"
Oct 30 00:33:19.677166 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.661577    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-be1f39-dockercfg-bpjhw\""
Oct 30 00:33:19.677166 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.661744    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-be1f39\""
Oct 30 00:33:19.729049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.707468    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg48v\" (UniqueName: \"kubernetes.io/projected/1910466b-ea6b-4193-8242-e117d64a1f3e-kube-api-access-kg48v\") pod \"payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5\" (UID: \"1910466b-ea6b-4193-8242-e117d64a1f3e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"
Oct 30 00:33:19.729049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.707653    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1910466b-ea6b-4193-8242-e117d64a1f3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5\" (UID: \"1910466b-ea6b-4193-8242-e117d64a1f3e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"
Oct 30 00:33:19.729049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.710083    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1910466b-ea6b-4193-8242-e117d64a1f3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5\" (UID: \"1910466b-ea6b-4193-8242-e117d64a1f3e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"
Oct 30 00:33:19.921123 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.914851    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"]
Oct 30 00:33:19.936292 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.922019    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg48v\" (UniqueName: \"kubernetes.io/projected/1910466b-ea6b-4193-8242-e117d64a1f3e-kube-api-access-kg48v\") pod \"payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5\" (UID: \"1910466b-ea6b-4193-8242-e117d64a1f3e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"
Oct 30 00:33:19.965617 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.965581    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"
Oct 30 00:33:19.968326 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.968053    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"]
Oct 30 00:33:19.969257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.969227    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 00:33:19.969363 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.969282    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" podUID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 00:33:19.972687 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:19.972663    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"
Oct 30 00:33:20.076140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.076013    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8xr5\" (UniqueName: \"kubernetes.io/projected/1228f328-ccb3-4ed2-b77a-5f8018f50e77-kube-api-access-x8xr5\") pod \"payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq\" (UID: \"1228f328-ccb3-4ed2-b77a-5f8018f50e77\") " pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"
Oct 30 00:33:20.076724 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.076364    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1228f328-ccb3-4ed2-b77a-5f8018f50e77-plugins-config-volume\") pod \"payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq\" (UID: \"1228f328-ccb3-4ed2-b77a-5f8018f50e77\") " pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"
Oct 30 00:33:20.178270 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.178197    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8xr5\" (UniqueName: \"kubernetes.io/projected/1228f328-ccb3-4ed2-b77a-5f8018f50e77-kube-api-access-x8xr5\") pod \"payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq\" (UID: \"1228f328-ccb3-4ed2-b77a-5f8018f50e77\") " pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"
Oct 30 00:33:20.178414 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.178280    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1228f328-ccb3-4ed2-b77a-5f8018f50e77-plugins-config-volume\") pod \"payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq\" (UID: \"1228f328-ccb3-4ed2-b77a-5f8018f50e77\") " pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"
Oct 30 00:33:20.179113 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.179083    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1228f328-ccb3-4ed2-b77a-5f8018f50e77-plugins-config-volume\") pod \"payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq\" (UID: \"1228f328-ccb3-4ed2-b77a-5f8018f50e77\") " pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"
Oct 30 00:33:20.199742 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.199713    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8xr5\" (UniqueName: \"kubernetes.io/projected/1228f328-ccb3-4ed2-b77a-5f8018f50e77-kube-api-access-x8xr5\") pod \"payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq\" (UID: \"1228f328-ccb3-4ed2-b77a-5f8018f50e77\") " pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"
Oct 30 00:33:20.286072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.286040    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"
Oct 30 00:33:20.319822 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.319786    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"]
Oct 30 00:33:20.320402 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:20.320364    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1910466b_ea6b_4193_8242_e117d64a1f3e.slice/crio-de1524e899714fc4feec10b88820ce96f61489cba790118d009e068053e1e2c0 WatchSource:0}: Error finding container de1524e899714fc4feec10b88820ce96f61489cba790118d009e068053e1e2c0: Status 404 returned error can't find the container with id de1524e899714fc4feec10b88820ce96f61489cba790118d009e068053e1e2c0
Oct 30 00:33:20.545049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.521355    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5" event={"ID":"1910466b-ea6b-4193-8242-e117d64a1f3e","Type":"ContainerStarted","Data":"57fb075e5020dc81992eb0b616414657ed784bf542b6bd85b6719fcea25a12e7"}
Oct 30 00:33:20.545049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.521418    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5" event={"ID":"1910466b-ea6b-4193-8242-e117d64a1f3e","Type":"ContainerStarted","Data":"de1524e899714fc4feec10b88820ce96f61489cba790118d009e068053e1e2c0"}
Oct 30 00:33:20.545049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.522469    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"
Oct 30 00:33:20.663251 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:20.663208    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1228f328_ccb3_4ed2_b77a_5f8018f50e77.slice/crio-1b605cebd8fa7208e03ecaae72ee5a57f38ac7a7bcb39d21b1d374ef822afa77 WatchSource:0}: Error finding container 1b605cebd8fa7208e03ecaae72ee5a57f38ac7a7bcb39d21b1d374ef822afa77: Status 404 returned error can't find the container with id 1b605cebd8fa7208e03ecaae72ee5a57f38ac7a7bcb39d21b1d374ef822afa77
Oct 30 00:33:20.667867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.665327    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5" podStartSLOduration=1.665308854 podStartE2EDuration="1.665308854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:20.570197592 +0000 UTC m=+1710.095491469" watchObservedRunningTime="2026-10-30 00:33:20.665308854 +0000 UTC m=+1710.190602734"
Oct 30 00:33:20.667867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.667695    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"]
Oct 30 00:33:20.944891 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.944439    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 00:33:20.944891 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:20.944505    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" podUID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 00:33:21.570577 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:21.570536    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq" event={"ID":"1228f328-ccb3-4ed2-b77a-5f8018f50e77","Type":"ContainerStarted","Data":"c3f3b684869e222d749bf83c40f4ce2c566360c7f4e2f3eb8d84ea9f7e30d0b9"}
Oct 30 00:33:21.571079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:21.570604    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"
Oct 30 00:33:21.571079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:21.570629    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq" event={"ID":"1228f328-ccb3-4ed2-b77a-5f8018f50e77","Type":"ContainerStarted","Data":"1b605cebd8fa7208e03ecaae72ee5a57f38ac7a7bcb39d21b1d374ef822afa77"}
Oct 30 00:33:21.606533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:21.606491    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq" podStartSLOduration=2.606477601 podStartE2EDuration="2.606477601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:21.605301356 +0000 UTC m=+1711.130595235" watchObservedRunningTime="2026-10-30 00:33:21.606477601 +0000 UTC m=+1711.131771503"
Oct 30 00:33:21.938533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:21.938376    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 00:33:21.938533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:21.938438    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" podUID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 00:33:22.945047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:22.938436    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 00:33:22.945047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:22.938503    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" podUID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 00:33:23.938470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:23.938429    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 00:33:23.938655 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:23.938499    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" podUID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 00:33:24.545392 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:24.545282    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:33:24.938509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:24.938411    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 00:33:24.943205 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:24.943165    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" podUID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 00:33:26.161050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:26.155120    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:26.237119 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:26.235819    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:26.383050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:26.350225    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:26.476180 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:26.446121    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:33:26.577050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:26.572145    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:26.577050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:26.575929    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:26.612053 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:26.607109    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:26.650894 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:26.650849    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:26.678068 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:26.677557    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:26.692052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:26.687577    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:27.056225 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.056189    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:33:27.056540 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.056506    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" podUID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:27.239289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.239244    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:33:27.263425 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.263393    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:33:27.486109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.483127    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:27.486109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.484405    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:33:27.507081 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.496340    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:27.507081 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.497083    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:27.507458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.507438    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:33:27.542049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.512658    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:27.542049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.513307    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:33:27.581048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.576380    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:27.581048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.577559    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:27.581317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.581055    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:33:27.581317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.581112    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:33:27.643263 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.639648    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:27.643263 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.640151    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:33:27.659060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.652280    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:33:27.943104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.678651    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:28.112657 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.692248    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:28.150050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:27.938103    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:33:28.239318 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:28.239017    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:33:28.239318 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:28.239125    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:33:28.264096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:28.240851    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:33:28.530730 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:28.516574    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 00:33:28.530730 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:28.518776    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:28.588048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:28.587396    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:28.611050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:28.588984    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:28.847108 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:28.838492    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 00:33:28.990083 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:28.940432    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 00:33:29.121732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.121634    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:33:29.151135 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.140013    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:33:29.312202 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.312132    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:29.453355 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.352809    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:33:29.545201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.545150    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:29.560055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.545685    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:33:29.568057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.547216    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"]
Oct 30 00:33:29.603455 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.591991    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:29.603455 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.593517    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.628812 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.624226    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-ecb34a-openshift-default-dockercfg-2wvdt\""
Oct 30 00:33:29.644155 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.642909    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:29.756071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.740430    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-socket\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.756071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.740582    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-certs\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.756071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.740655    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-credential-socket\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.756071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.740685    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-podinfo\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.756071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.740727    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-envoy\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.756071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.740788    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-token\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.756071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.740815    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-data\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.756071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.740918    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istiod-ca-cert\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.756071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.741128    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpfds\" (UniqueName: \"kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-kube-api-access-kpfds\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.857330 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.857293    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-certs\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.857616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.857592    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-credential-socket\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.857760 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.857741    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-podinfo\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.857897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.857880    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-envoy\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.858063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.858046    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-token\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.858208 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.858189    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-data\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.858385 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.858365    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istiod-ca-cert\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.862484 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.858609    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpfds\" (UniqueName: \"kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-kube-api-access-kpfds\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.862773 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.862748    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-socket\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.863585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.863560    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-socket\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.863989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.863969    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-certs\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.864427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.864406    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-credential-socket\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.865529 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.865505    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-data\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.866337 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.866317    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istiod-ca-cert\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:29.889749 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.887182    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:33:29.948077 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:29.944146    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"]
Oct 30 00:33:30.026367 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.026294    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-podinfo\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:30.027074 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.027051    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-token\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:30.075172 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.075107    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpfds\" (UniqueName: \"kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-kube-api-access-kpfds\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:30.140004 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.139977    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:33:30.149689 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.149656    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:33:30.213105 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.211775    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:33:30.217369 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.216910    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:33:30.226105 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.222090    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:33:30.299010 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.298931    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-envoy\") pod \"e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") " pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:30.329253 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.329158    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:33:30.337668 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.336822    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:33:30.344214 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.344164    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:33:30.348344 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.348046    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:33:30.531133 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.531096    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:31.000066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:30.999994    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"]
Oct 30 00:33:31.000178 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:31.000000    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66cdbd4_e6a1_44e2_b459_c1502f57225a.slice/crio-acdefda43f741831f2844101b176a765f001ccaed1bb845c2f99e73c770e11bd WatchSource:0}: Error finding container acdefda43f741831f2844101b176a765f001ccaed1bb845c2f99e73c770e11bd: Status 404 returned error can't find the container with id acdefda43f741831f2844101b176a765f001ccaed1bb845c2f99e73c770e11bd
Oct 30 00:33:31.003098 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.003065    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:31.003241 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.003149    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:31.003241 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.003215    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:31.545097 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.545060    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" event={"ID":"b66cdbd4-e6a1-44e2-b459-c1502f57225a","Type":"ContainerStarted","Data":"ed24d7c957fc8094b9eec347e8b9e6bedb792ad631904b3647b80e6034f41684"}
Oct 30 00:33:31.545560 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.545166    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" event={"ID":"b66cdbd4-e6a1-44e2-b459-c1502f57225a","Type":"ContainerStarted","Data":"acdefda43f741831f2844101b176a765f001ccaed1bb845c2f99e73c770e11bd"}
Oct 30 00:33:31.608731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.608673    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" podStartSLOduration=2.608655186 podStartE2EDuration="2.608655186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:31.595262851 +0000 UTC m=+1721.120556730" watchObservedRunningTime="2026-10-30 00:33:31.608655186 +0000 UTC m=+1721.133949064"
Oct 30 00:33:31.655889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.655858    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"]
Oct 30 00:33:31.668937 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.668899    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.683997 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.683967    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-415d24-openshift-default-dockercfg-fxwjn\""
Oct 30 00:33:31.687417 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.687374    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"]
Oct 30 00:33:31.768413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.768377    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0fec19-a16c-40a1-8840-66022d000e75-istiod-ca-cert\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.768624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.768594    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-istio-token\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.768745 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.768661    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-envoy\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.768745 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.768697    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-certs\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.768874 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.768784    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-socket\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.768995 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.768942    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55p9\" (UniqueName: \"kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-kube-api-access-f55p9\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.769133 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.769076    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-credential-socket\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.769302 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.769263    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-data\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.769389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.769360    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0fec19-a16c-40a1-8840-66022d000e75-istio-podinfo\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.871574 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.871463    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-socket\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.871753 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.871691    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55p9\" (UniqueName: \"kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-kube-api-access-f55p9\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.871829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.871777    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-credential-socket\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.871897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.871856    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-data\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.871897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.871879    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0fec19-a16c-40a1-8840-66022d000e75-istio-podinfo\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.871897 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.871896    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0fec19-a16c-40a1-8840-66022d000e75-istiod-ca-cert\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.872127 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.871940    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-socket\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.872127 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.871978    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-istio-token\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.872127 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.872003    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-envoy\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.872127 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.872038    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-certs\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.872362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.872281    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-data\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.872362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.872327    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-certs\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.873506 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.873480    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0fec19-a16c-40a1-8840-66022d000e75-istiod-ca-cert\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.875216 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.875192    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-envoy\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.875356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.875338    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0fec19-a16c-40a1-8840-66022d000e75-istio-podinfo\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.875470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.875449    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-credential-socket\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.894633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.894603    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55p9\" (UniqueName: \"kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-kube-api-access-f55p9\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.896246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.896219    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-istio-token\") pod \"e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") " pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:31.991400 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:31.991363    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:32.340724 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.340692    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"]
Oct 30 00:33:32.341792 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:32.341761    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0fec19_a16c_40a1_8840_66022d000e75.slice/crio-ec14f94b420c63e2cb805694d3f80ec9b686c4243187a0964b5609a66f2fde01 WatchSource:0}: Error finding container ec14f94b420c63e2cb805694d3f80ec9b686c4243187a0964b5609a66f2fde01: Status 404 returned error can't find the container with id ec14f94b420c63e2cb805694d3f80ec9b686c4243187a0964b5609a66f2fde01
Oct 30 00:33:32.345846 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.345803    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:32.345978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.345898    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:32.345978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.345937    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:32.539549 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.539516    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:32.555072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.548095    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:32.564168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.564137    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" event={"ID":"5c0fec19-a16c-40a1-8840-66022d000e75","Type":"ContainerStarted","Data":"227177ae1363bdfb5b2019cfc6fc273f4814a4fa1a8ab60ed83cc318a187b931"}
Oct 30 00:33:32.564390 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.564192    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" event={"ID":"5c0fec19-a16c-40a1-8840-66022d000e75","Type":"ContainerStarted","Data":"ec14f94b420c63e2cb805694d3f80ec9b686c4243187a0964b5609a66f2fde01"}
Oct 30 00:33:32.564621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.564599    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:32.565334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.565315    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:33:32.587912 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.587848    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"
Oct 30 00:33:32.588133 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.588062    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"
Oct 30 00:33:32.757073 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.756995    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" podStartSLOduration=1.756974433 podStartE2EDuration="1.756974433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:32.721405523 +0000 UTC m=+1722.246699401" watchObservedRunningTime="2026-10-30 00:33:32.756974433 +0000 UTC m=+1722.282268310"
Oct 30 00:33:32.857371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.857261    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"]
Oct 30 00:33:32.930484 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.930454    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"]
Oct 30 00:33:32.932119 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.932078    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"]
Oct 30 00:33:32.935157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.930756    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:32.945100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.945074    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"]
Oct 30 00:33:32.945607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.945586    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:32.991893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:32.991868    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:33.008145 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008113    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.008329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008161    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.008329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008203    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.008329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008235    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.008329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008304    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.008329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008327    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrkbz\" (UniqueName: \"kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-kube-api-access-vrkbz\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.008615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008391    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9fcd\" (UniqueName: \"kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-kube-api-access-q9fcd\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.008615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008444    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.008615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008477    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59de270-9763-49fa-8efc-625d58575e48-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.008615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008601    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.008818 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008667    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.008818 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008715    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.008818 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008759    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.008969 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008820    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.008969 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008863    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59de270-9763-49fa-8efc-625d58575e48-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.009135 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.008970    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.009135 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.009064    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.009135 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.009098    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14e3e15-a207-4782-9905-befa94ac0ddc-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.109991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.109903    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.109991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.109952    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.109991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.109988    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.110306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110014    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.110306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110115    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.110306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110152    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59de270-9763-49fa-8efc-625d58575e48-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.110306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110238    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.110306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110293    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.110619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110328    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14e3e15-a207-4782-9905-befa94ac0ddc-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.110619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110358    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.110619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110389    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.110619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110400    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.110619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110426    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.110619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110437    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.110619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110436    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.110619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110509    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.110619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110590    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.110619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110620    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrkbz\" (UniqueName: \"kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-kube-api-access-vrkbz\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.111230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110640    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.111230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110705    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.111230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110717    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9fcd\" (UniqueName: \"kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-kube-api-access-q9fcd\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.111230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110786    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.111230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110820    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59de270-9763-49fa-8efc-625d58575e48-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.111230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.110958    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.111230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.111121    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14e3e15-a207-4782-9905-befa94ac0ddc-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.111524 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.111240    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.111835 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.111811    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59de270-9763-49fa-8efc-625d58575e48-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.113289 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.113269    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.113518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.113352    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.114111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.114019    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59de270-9763-49fa-8efc-625d58575e48-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.114111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.114094    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.114582 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.114559    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.151366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.151329    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrkbz\" (UniqueName: \"kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-kube-api-access-vrkbz\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.166092 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.165882    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9fcd\" (UniqueName: \"kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-kube-api-access-q9fcd\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.172360 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.172330    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.174894 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.174863    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.264350 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.264313    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:33.289002 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.288974    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:33.601131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.601103    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"]
Oct 30 00:33:33.602352 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:33.602326    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59de270_9763_49fa_8efc_625d58575e48.slice/crio-0230b2b5e1067ca3f8e2b3edea5149ce1034a99a267d70e6eafd1cae322ff891 WatchSource:0}: Error finding container 0230b2b5e1067ca3f8e2b3edea5149ce1034a99a267d70e6eafd1cae322ff891: Status 404 returned error can't find the container with id 0230b2b5e1067ca3f8e2b3edea5149ce1034a99a267d70e6eafd1cae322ff891
Oct 30 00:33:33.606182 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.606143    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:33.606302 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.606227    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:33.606302 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.606268    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:33.642076 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.641723    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"]
Oct 30 00:33:33.647878 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.647842    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:33.648077 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.648056    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:33.648234 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:33.648217    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:33:33.961958 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:33:33.961794    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:33:34.001421 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:34.001395    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:34.591450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:34.591408    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd" event={"ID":"e14e3e15-a207-4782-9905-befa94ac0ddc","Type":"ContainerStarted","Data":"9a3fa24bdd8e3e163a0f7d91611f2dd5f812bf75e1a8b7b9fbfa8c79128ed5e9"}
Oct 30 00:33:34.591775 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:34.591753    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd" event={"ID":"e14e3e15-a207-4782-9905-befa94ac0ddc","Type":"ContainerStarted","Data":"fbaadad22e3b1f1fc21617e4b1faa130932c5de4abeca442128abe4e22d7041e"}
Oct 30 00:33:34.598362 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:34.598325    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg" event={"ID":"d59de270-9763-49fa-8efc-625d58575e48","Type":"ContainerStarted","Data":"35b3816932b30c2af46a88dc1ab8a87f7c6a5fd5d820114ca5daeb8377be4cdf"}
Oct 30 00:33:34.599736 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:34.599707    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg" event={"ID":"d59de270-9763-49fa-8efc-625d58575e48","Type":"ContainerStarted","Data":"0230b2b5e1067ca3f8e2b3edea5149ce1034a99a267d70e6eafd1cae322ff891"}
Oct 30 00:33:34.600191 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:34.600171    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:34.600389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:34.600377    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:33:34.708074 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:34.695646    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd" podStartSLOduration=2.695623963 podStartE2EDuration="2.695623963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:34.657497532 +0000 UTC m=+1724.182791408" watchObservedRunningTime="2026-10-30 00:33:34.695623963 +0000 UTC m=+1724.220917843"
Oct 30 00:33:34.889117 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:34.886045    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg" podStartSLOduration=2.886007296 podStartE2EDuration="2.886007296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:34.800346489 +0000 UTC m=+1724.325640366" watchObservedRunningTime="2026-10-30 00:33:34.886007296 +0000 UTC m=+1724.411301174"
Oct 30 00:33:35.279672 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:35.279642    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:35.287195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:35.287160    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:33:35.287463 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:35.287432    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg" podUID="d59de270-9763-49fa-8efc-625d58575e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 00:33:35.293085 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:35.292659    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:35.298225 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:35.298192    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 00:33:35.298348 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:35.298248    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd" podUID="e14e3e15-a207-4782-9905-befa94ac0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 00:33:35.298423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:35.298398    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"]
Oct 30 00:33:35.304075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:35.304049    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"]
Oct 30 00:33:36.299916 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:36.299302    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 00:33:36.299916 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:36.299360    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd" podUID="e14e3e15-a207-4782-9905-befa94ac0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 00:33:36.309853 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:36.309699    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:33:36.309853 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:36.309752    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg" podUID="d59de270-9763-49fa-8efc-625d58575e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 00:33:36.630283 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:36.630185    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 00:33:36.652158 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:36.652119    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:33:37.265455 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:37.265412    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:33:37.265751 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:37.265718    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg" podUID="d59de270-9763-49fa-8efc-625d58575e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 00:33:37.290317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:37.290288    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 00:33:37.290552 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:37.290527    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd" podUID="e14e3e15-a207-4782-9905-befa94ac0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 00:33:38.274048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:38.273581    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:33:38.274048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:38.273671    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg" podUID="d59de270-9763-49fa-8efc-625d58575e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 00:33:38.291759 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:38.291721    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 00:33:38.292206 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:38.292140    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd" podUID="e14e3e15-a207-4782-9905-befa94ac0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 00:33:39.274677 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:39.274643    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:33:39.275342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:39.275288    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg" podUID="d59de270-9763-49fa-8efc-625d58575e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 00:33:39.310443 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:39.310412    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 00:33:39.310727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:39.310692    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd" podUID="e14e3e15-a207-4782-9905-befa94ac0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 00:33:40.264720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:40.264681    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:33:40.265041 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:40.264992    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg" podUID="d59de270-9763-49fa-8efc-625d58575e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 00:33:40.289514 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:40.289466    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 00:33:40.290124 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:40.290090    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd" podUID="e14e3e15-a207-4782-9905-befa94ac0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 00:33:41.265498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:41.265455    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:33:41.265707 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:41.265545    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg" podUID="d59de270-9763-49fa-8efc-625d58575e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 00:33:41.289797 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:41.289757    4441 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 00:33:41.290205 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:41.289841    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd" podUID="e14e3e15-a207-4782-9905-befa94ac0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 00:33:42.296247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.294116    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:42.296247 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.295394    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:42.297322 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.297297    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:33:42.331780 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.331737    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:42.333856 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.333422    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:42.334870 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.334840    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:33:42.477899 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.477861    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"]
Oct 30 00:33:42.487068 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.487015    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:42.502218 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.502179    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-be1f39-serving-cert\""
Oct 30 00:33:42.505551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.505526    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"]
Oct 30 00:33:42.562342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.562249    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a472f11-984e-4097-9017-b9f8e7434853-openshift-service-ca\") pod \"maas-api-e2e-explicit-be1f39-65cddf6898-899gd\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:42.562522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.562362    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-metrics-certs\") pod \"maas-api-e2e-explicit-be1f39-65cddf6898-899gd\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:42.562522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.562430    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-maas-api-tls\") pod \"maas-api-e2e-explicit-be1f39-65cddf6898-899gd\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:42.562739 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.562708    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnzr\" (UniqueName: \"kubernetes.io/projected/6a472f11-984e-4097-9017-b9f8e7434853-kube-api-access-rfnzr\") pod \"maas-api-e2e-explicit-be1f39-65cddf6898-899gd\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:42.664103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.664047    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a472f11-984e-4097-9017-b9f8e7434853-openshift-service-ca\") pod \"maas-api-e2e-explicit-be1f39-65cddf6898-899gd\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:42.664103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.664107    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-metrics-certs\") pod \"maas-api-e2e-explicit-be1f39-65cddf6898-899gd\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:42.664373 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.664126    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-maas-api-tls\") pod \"maas-api-e2e-explicit-be1f39-65cddf6898-899gd\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:42.664373 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.664355    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnzr\" (UniqueName: \"kubernetes.io/projected/6a472f11-984e-4097-9017-b9f8e7434853-kube-api-access-rfnzr\") pod \"maas-api-e2e-explicit-be1f39-65cddf6898-899gd\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:42.665578 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.665549    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a472f11-984e-4097-9017-b9f8e7434853-openshift-service-ca\") pod \"maas-api-e2e-explicit-be1f39-65cddf6898-899gd\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:42.666063 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:33:42.666042    4441 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-serving-cert: secret "maas-api-e2e-explicit-be1f39-serving-cert" not found
Oct 30 00:33:42.666322 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:33:42.666307    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-maas-api-tls podName:6a472f11-984e-4097-9017-b9f8e7434853 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:43.166285071 +0000 UTC m=+1732.691578943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-maas-api-tls") pod "maas-api-e2e-explicit-be1f39-65cddf6898-899gd" (UID: "6a472f11-984e-4097-9017-b9f8e7434853") : secret "maas-api-e2e-explicit-be1f39-serving-cert" not found
Oct 30 00:33:42.668491 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.668454    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-metrics-certs\") pod \"maas-api-e2e-explicit-be1f39-65cddf6898-899gd\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:42.691008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:42.688148    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnzr\" (UniqueName: \"kubernetes.io/projected/6a472f11-984e-4097-9017-b9f8e7434853-kube-api-access-rfnzr\") pod \"maas-api-e2e-explicit-be1f39-65cddf6898-899gd\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:43.174990 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:43.174953    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-maas-api-tls\") pod \"maas-api-e2e-explicit-be1f39-65cddf6898-899gd\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:43.177608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:43.177582    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-maas-api-tls\") pod \"maas-api-e2e-explicit-be1f39-65cddf6898-899gd\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:43.414997 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:43.414969    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:43.737246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:43.737219    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"]
Oct 30 00:33:43.738478 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:43.738450    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a472f11_984e_4097_9017_b9f8e7434853.slice/crio-5c35522efff41642b6ead3c4ce03b40f1eb65ba8f2046f1ad6b0e0abf9f44c6f WatchSource:0}: Error finding container 5c35522efff41642b6ead3c4ce03b40f1eb65ba8f2046f1ad6b0e0abf9f44c6f: Status 404 returned error can't find the container with id 5c35522efff41642b6ead3c4ce03b40f1eb65ba8f2046f1ad6b0e0abf9f44c6f
Oct 30 00:33:44.114665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:44.114640    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd" event={"ID":"6a472f11-984e-4097-9017-b9f8e7434853","Type":"ContainerStarted","Data":"5c35522efff41642b6ead3c4ce03b40f1eb65ba8f2046f1ad6b0e0abf9f44c6f"}
Oct 30 00:33:45.123566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:45.123528    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd" event={"ID":"6a472f11-984e-4097-9017-b9f8e7434853","Type":"ContainerStarted","Data":"7a671a14d31e8e2b72fbc77e0e9f3fadb892801802c6285e98884ed58c339c82"}
Oct 30 00:33:45.123968 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:45.123632    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:45.166005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:45.165951    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd" podStartSLOduration=2.865787688 podStartE2EDuration="3.165936772s" totalImagesPullingTime="300.149084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:45.16078852 +0000 UTC m=+1734.686082397" watchObservedRunningTime="2026-10-30 00:33:45.165936772 +0000 UTC m=+1734.691230684"
Oct 30 00:33:50.409496 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.409456    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main" containerID="cri-o://3b5cb819663519d74a834e7f07863712fa4eadfef944ddb760b4a83225a9cb31" gracePeriod=45
Oct 30 00:33:50.409936 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.409496    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main" containerID="cri-o://ae0c003b1e09213693848ae6216c287f8e425fd1d61c3a0c70e61aeecfe50d0a" gracePeriod=45
Oct 30 00:33:50.756474 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.756451    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:33:50.760213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.760191    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:33:50.843708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.843668    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-tmp-dir\") pod \"736f577a-2a21-4448-8c1e-bb71782053ec\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") "
Oct 30 00:33:50.844058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.844015    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-kube-api-access-t9whp\" (UniqueName: \"kubernetes.io/projected/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-kube-api-access-t9whp\") pod \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") "
Oct 30 00:33:50.844233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.844218    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tmp-dir\") pod \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") "
Oct 30 00:33:50.844358 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.844343    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736f577a-2a21-4448-8c1e-bb71782053ec-kube-api-access-rr9db\" (UniqueName: \"kubernetes.io/projected/736f577a-2a21-4448-8c1e-bb71782053ec-kube-api-access-rr9db\") pod \"736f577a-2a21-4448-8c1e-bb71782053ec\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") "
Oct 30 00:33:50.844506 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.844492    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-dshm\") pod \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") "
Oct 30 00:33:50.844726 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.844711    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-dshm\") pod \"736f577a-2a21-4448-8c1e-bb71782053ec\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") "
Oct 30 00:33:50.844842 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.844827    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-model-cache\") pod \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") "
Oct 30 00:33:50.844992 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.844980    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-home\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-home\") pod \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") "
Oct 30 00:33:50.845164 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.845145    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736f577a-2a21-4448-8c1e-bb71782053ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/736f577a-2a21-4448-8c1e-bb71782053ec-tls-certs\") pod \"736f577a-2a21-4448-8c1e-bb71782053ec\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") "
Oct 30 00:33:50.845295 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.845282    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-model-cache\") pod \"736f577a-2a21-4448-8c1e-bb71782053ec\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") "
Oct 30 00:33:50.845542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.845518    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tls-certs\") pod \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\" (UID: \"e4829f2a-b4dc-413e-b235-66bc5c0edaaf\") "
Oct 30 00:33:50.845668 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.845620    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-home\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-home\") pod \"736f577a-2a21-4448-8c1e-bb71782053ec\" (UID: \"736f577a-2a21-4448-8c1e-bb71782053ec\") "
Oct 30 00:33:50.846977 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.846953    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-home" pod "736f577a-2a21-4448-8c1e-bb71782053ec" (UID: "736f577a-2a21-4448-8c1e-bb71782053ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:33:50.847124 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.846962    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tmp-dir" pod "e4829f2a-b4dc-413e-b235-66bc5c0edaaf" (UID: "e4829f2a-b4dc-413e-b235-66bc5c0edaaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:33:50.847535 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.847515    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-model-cache" pod "736f577a-2a21-4448-8c1e-bb71782053ec" (UID: "736f577a-2a21-4448-8c1e-bb71782053ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:33:50.847773 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.847747    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-model-cache" pod "e4829f2a-b4dc-413e-b235-66bc5c0edaaf" (UID: "e4829f2a-b4dc-413e-b235-66bc5c0edaaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:33:50.848013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.847994    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-home" pod "e4829f2a-b4dc-413e-b235-66bc5c0edaaf" (UID: "e4829f2a-b4dc-413e-b235-66bc5c0edaaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:33:50.848788 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.848763    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-tmp-dir" pod "736f577a-2a21-4448-8c1e-bb71782053ec" (UID: "736f577a-2a21-4448-8c1e-bb71782053ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:33:50.849907 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.849886    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-dshm" pod "736f577a-2a21-4448-8c1e-bb71782053ec" (UID: "736f577a-2a21-4448-8c1e-bb71782053ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:33:50.850207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.850165    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-dshm" pod "e4829f2a-b4dc-413e-b235-66bc5c0edaaf" (UID: "e4829f2a-b4dc-413e-b235-66bc5c0edaaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:33:50.851546 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.851523    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736f577a-2a21-4448-8c1e-bb71782053ec-tls-certs" pod "736f577a-2a21-4448-8c1e-bb71782053ec" (UID: "736f577a-2a21-4448-8c1e-bb71782053ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:33:50.852338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.852310    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736f577a-2a21-4448-8c1e-bb71782053ec-kube-api-access-rr9db" pod "736f577a-2a21-4448-8c1e-bb71782053ec" (UID: "736f577a-2a21-4448-8c1e-bb71782053ec"). InnerVolumeSpecName "kube-api-access-rr9db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:33:50.852448 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.852405    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tls-certs" pod "e4829f2a-b4dc-413e-b235-66bc5c0edaaf" (UID: "e4829f2a-b4dc-413e-b235-66bc5c0edaaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:33:50.853762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.853733    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-kube-api-access-t9whp" pod "e4829f2a-b4dc-413e-b235-66bc5c0edaaf" (UID: "e4829f2a-b4dc-413e-b235-66bc5c0edaaf"). InnerVolumeSpecName "kube-api-access-t9whp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:33:50.947893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.947858    4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-tmp-dir\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:33:50.947893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.947891    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9whp\" (UniqueName: \"kubernetes.io/projected/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-kube-api-access-t9whp\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:33:50.948093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.947903    4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tmp-dir\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:33:50.948093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.947912    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr9db\" (UniqueName: \"kubernetes.io/projected/736f577a-2a21-4448-8c1e-bb71782053ec-kube-api-access-rr9db\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:33:50.948093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.947921    4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-dshm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:33:50.948093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.947929    4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-dshm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:33:50.948093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.947937    4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-model-cache\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:33:50.948093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.947946    4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-home\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:33:50.948093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.947953    4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736f577a-2a21-4448-8c1e-bb71782053ec-tls-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:33:50.948093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.947960    4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-model-cache\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:33:50.948093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.947967    4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4829f2a-b4dc-413e-b235-66bc5c0edaaf-tls-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:33:50.948093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:50.947974    4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736f577a-2a21-4448-8c1e-bb71782053ec-home\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:33:51.130615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.130579    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"]
Oct 30 00:33:51.144315 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.144289    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:33:51.151817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.151787    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"]
Oct 30 00:33:51.178569 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.178540    4441 generic.go:356] "Generic (PLEG): container finished" podID="736f577a-2a21-4448-8c1e-bb71782053ec" containerID="ae0c003b1e09213693848ae6216c287f8e425fd1d61c3a0c70e61aeecfe50d0a" exitCode=0
Oct 30 00:33:51.178743 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.178571    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"
Oct 30 00:33:51.178743 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.178618    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" event={"ID":"736f577a-2a21-4448-8c1e-bb71782053ec","Type":"ContainerDied","Data":"ae0c003b1e09213693848ae6216c287f8e425fd1d61c3a0c70e61aeecfe50d0a"}
Oct 30 00:33:51.178743 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.178661    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw" event={"ID":"736f577a-2a21-4448-8c1e-bb71782053ec","Type":"ContainerDied","Data":"a6614b32bec3823f454b3ad10b3ea14929297b157096b7d8f0ff17b200024d05"}
Oct 30 00:33:51.178743 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.178686    4441 scope.go:122] "RemoveContainer" containerID="ae0c003b1e09213693848ae6216c287f8e425fd1d61c3a0c70e61aeecfe50d0a"
Oct 30 00:33:51.180462 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.180325    4441 generic.go:356] "Generic (PLEG): container finished" podID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerID="3b5cb819663519d74a834e7f07863712fa4eadfef944ddb760b4a83225a9cb31" exitCode=0
Oct 30 00:33:51.180462 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.180387    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"
Oct 30 00:33:51.184066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.180674    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" event={"ID":"e4829f2a-b4dc-413e-b235-66bc5c0edaaf","Type":"ContainerDied","Data":"3b5cb819663519d74a834e7f07863712fa4eadfef944ddb760b4a83225a9cb31"}
Oct 30 00:33:51.184066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.180719    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l" event={"ID":"e4829f2a-b4dc-413e-b235-66bc5c0edaaf","Type":"ContainerDied","Data":"18209faf4b08e58d7022db1d1f9acd08d062aa1d98593091d04bc596fe5361fe"}
Oct 30 00:33:51.185477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.185455    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"]
Oct 30 00:33:51.192124 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.192108    4441 scope.go:122] "RemoveContainer" containerID="ae0c003b1e09213693848ae6216c287f8e425fd1d61c3a0c70e61aeecfe50d0a"
Oct 30 00:33:51.192398 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:33:51.192380    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0c003b1e09213693848ae6216c287f8e425fd1d61c3a0c70e61aeecfe50d0a\": container with ID starting with ae0c003b1e09213693848ae6216c287f8e425fd1d61c3a0c70e61aeecfe50d0a not found: ID does not exist" containerID="ae0c003b1e09213693848ae6216c287f8e425fd1d61c3a0c70e61aeecfe50d0a"
Oct 30 00:33:51.192475 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.192408    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0c003b1e09213693848ae6216c287f8e425fd1d61c3a0c70e61aeecfe50d0a"} err="failed to get container status \"ae0c003b1e09213693848ae6216c287f8e425fd1d61c3a0c70e61aeecfe50d0a\": rpc error: code = NotFound desc = could not find container \"ae0c003b1e09213693848ae6216c287f8e425fd1d61c3a0c70e61aeecfe50d0a\": container with ID starting with ae0c003b1e09213693848ae6216c287f8e425fd1d61c3a0c70e61aeecfe50d0a not found: ID does not exist"
Oct 30 00:33:51.192475 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.192430    4441 scope.go:122] "RemoveContainer" containerID="3b5cb819663519d74a834e7f07863712fa4eadfef944ddb760b4a83225a9cb31"
Oct 30 00:33:51.202893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.202875    4441 scope.go:122] "RemoveContainer" containerID="3b5cb819663519d74a834e7f07863712fa4eadfef944ddb760b4a83225a9cb31"
Oct 30 00:33:51.203188 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:33:51.203170    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5cb819663519d74a834e7f07863712fa4eadfef944ddb760b4a83225a9cb31\": container with ID starting with 3b5cb819663519d74a834e7f07863712fa4eadfef944ddb760b4a83225a9cb31 not found: ID does not exist" containerID="3b5cb819663519d74a834e7f07863712fa4eadfef944ddb760b4a83225a9cb31"
Oct 30 00:33:51.203240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.203195    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5cb819663519d74a834e7f07863712fa4eadfef944ddb760b4a83225a9cb31"} err="failed to get container status \"3b5cb819663519d74a834e7f07863712fa4eadfef944ddb760b4a83225a9cb31\": rpc error: code = NotFound desc = could not find container \"3b5cb819663519d74a834e7f07863712fa4eadfef944ddb760b4a83225a9cb31\": container with ID starting with 3b5cb819663519d74a834e7f07863712fa4eadfef944ddb760b4a83225a9cb31 not found: ID does not exist"
Oct 30 00:33:51.319647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.319614    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"]
Oct 30 00:33:51.327621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.327596    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/distinct-w5-b1c718-kserve-74c4f7c6fd-7xn8l"]
Oct 30 00:33:51.353068 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.352994    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"]
Oct 30 00:33:51.359684 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:51.359658    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/premium-w5-b1c718-kserve-846d8676d7-9z4bw"]
Oct 30 00:33:52.614592 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:52.614555    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" path="/var/lib/kubelet/pods/736f577a-2a21-4448-8c1e-bb71782053ec/volumes"
Oct 30 00:33:52.615321 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:52.615297    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" path="/var/lib/kubelet/pods/e4829f2a-b4dc-413e-b235-66bc5c0edaaf/volumes"
Oct 30 00:33:56.152790 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:56.152743    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz" podUID="21bca81c-b78d-45a0-8c76-13b92029347e" containerName="payload-processing" containerID="cri-o://6996d4f1ff17ccc6899a9e762d0804c606cc548c0fc937cbca2ef2620378ef1d" gracePeriod=25
Oct 30 00:33:56.186760 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:56.186724    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp" podUID="7ffa771a-b9fc-481e-8a37-9831c31780a2" containerName="payload-pre-processing" containerID="cri-o://3d9167bd5a3d7dea9a28bda44846fa7d439da6b97d199c25655432f139856b07" gracePeriod=25
Oct 30 00:33:56.967328 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:56.967288    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"]
Oct 30 00:33:56.968431 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:56.968410    4441 cpu_manager.go:414] "containerMap: removing container" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main"
Oct 30 00:33:56.968479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:56.968434    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main"
Oct 30 00:33:56.968479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:56.968453    4441 cpu_manager.go:414] "containerMap: removing container" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main"
Oct 30 00:33:56.968479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:56.968461    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main"
Oct 30 00:33:56.968651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:56.968639    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4829f2a-b4dc-413e-b235-66bc5c0edaaf" containerName="main"
Oct 30 00:33:56.968686 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:56.968655    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="736f577a-2a21-4448-8c1e-bb71782053ec" containerName="main"
Oct 30 00:33:56.975206 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:56.975185    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:56.989813 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:56.989790    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"]
Oct 30 00:33:56.996518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:56.996494    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-ecb34a-serving-cert\""
Oct 30 00:33:57.009385 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.009359    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:57.009521 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.009407    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:57.009592 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.009564    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:57.009666 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.009645    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4tkn\" (UniqueName: \"kubernetes.io/projected/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-kube-api-access-j4tkn\") pod \"maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:57.111755 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.111723    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:57.111924 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.111776    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4tkn\" (UniqueName: \"kubernetes.io/projected/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-kube-api-access-j4tkn\") pod \"maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:57.111992 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.111951    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:57.112041 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.112008    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:57.112188 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:33:57.112169    4441 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-serving-cert: secret "maas-api-e2e-shared-b-w4-ecb34a-serving-cert" not found
Oct 30 00:33:57.112250 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:33:57.112239    4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-maas-api-tls podName:58e7b78b-01af-4965-a2ee-0f73c0bf2e58 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:57.612224913 +0000 UTC m=+1747.137518796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-maas-api-tls") pod "maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl" (UID: "58e7b78b-01af-4965-a2ee-0f73c0bf2e58") : secret "maas-api-e2e-shared-b-w4-ecb34a-serving-cert" not found
Oct 30 00:33:57.112381 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.112354    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:57.114498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.114478    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:57.135138 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.135105    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4tkn\" (UniqueName: \"kubernetes.io/projected/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-kube-api-access-j4tkn\") pod \"maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:57.620522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.620485    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:57.623002 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.622974    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:57.650624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.650595    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"]
Oct 30 00:33:57.902344 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:57.902217    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:58.227624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:58.227540    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"]
Oct 30 00:33:58.230348 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:33:58.230308    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e7b78b_01af_4965_a2ee_0f73c0bf2e58.slice/crio-cbbe777a38395deb8b09c9e4c7fb075ee453c361a5ad44285038381e6ec26be4 WatchSource:0}: Error finding container cbbe777a38395deb8b09c9e4c7fb075ee453c361a5ad44285038381e6ec26be4: Status 404 returned error can't find the container with id cbbe777a38395deb8b09c9e4c7fb075ee453c361a5ad44285038381e6ec26be4
Oct 30 00:33:58.243714 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:58.243683    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl" event={"ID":"58e7b78b-01af-4965-a2ee-0f73c0bf2e58","Type":"ContainerStarted","Data":"cbbe777a38395deb8b09c9e4c7fb075ee453c361a5ad44285038381e6ec26be4"}
Oct 30 00:33:59.255805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:59.255770    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl" event={"ID":"58e7b78b-01af-4965-a2ee-0f73c0bf2e58","Type":"ContainerStarted","Data":"bffc3730df340cb3f57ed25e0cf63096a08b18ad3283e1b404a8c20fa8486618"}
Oct 30 00:33:59.256286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:59.255992    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:33:59.347110 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:33:59.347049    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl" podStartSLOduration=3.013425506 podStartE2EDuration="3.347018248s" totalImagesPullingTime="333.592742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:59.343428359 +0000 UTC m=+1748.868722237" watchObservedRunningTime="2026-10-30 00:33:59.347018248 +0000 UTC m=+1748.872312125"
Oct 30 00:34:00.209705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:00.209660    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"]
Oct 30 00:34:00.216382 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:00.216337    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"
Oct 30 00:34:00.225495 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:00.225446    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"]
Oct 30 00:34:00.245444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:00.245142    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-f1268c\""
Oct 30 00:34:00.245444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:00.245342    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-f1268c-dockercfg-j69rr\""
Oct 30 00:34:00.309986 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:00.309545    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnw62\" (UniqueName: \"kubernetes.io/projected/2af359bf-d670-4f66-9dab-2a95b328705e-kube-api-access-qnw62\") pod \"payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j\" (UID: \"2af359bf-d670-4f66-9dab-2a95b328705e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"
Oct 30 00:34:00.309986 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:00.309636    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af359bf-d670-4f66-9dab-2a95b328705e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j\" (UID: \"2af359bf-d670-4f66-9dab-2a95b328705e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"
Oct 30 00:34:00.411416 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:00.411352    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af359bf-d670-4f66-9dab-2a95b328705e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j\" (UID: \"2af359bf-d670-4f66-9dab-2a95b328705e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"
Oct 30 00:34:00.411924 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:00.411838    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnw62\" (UniqueName: \"kubernetes.io/projected/2af359bf-d670-4f66-9dab-2a95b328705e-kube-api-access-qnw62\") pod \"payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j\" (UID: \"2af359bf-d670-4f66-9dab-2a95b328705e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"
Oct 30 00:34:00.412355 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:00.412327    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af359bf-d670-4f66-9dab-2a95b328705e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j\" (UID: \"2af359bf-d670-4f66-9dab-2a95b328705e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"
Oct 30 00:34:00.450064 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:00.449064    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnw62\" (UniqueName: \"kubernetes.io/projected/2af359bf-d670-4f66-9dab-2a95b328705e-kube-api-access-qnw62\") pod \"payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j\" (UID: \"2af359bf-d670-4f66-9dab-2a95b328705e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"
Oct 30 00:34:00.602532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:00.602497    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"
Oct 30 00:34:00.958251 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:00.946536    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"]
Oct 30 00:34:01.316843 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:01.316807    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j" event={"ID":"2af359bf-d670-4f66-9dab-2a95b328705e","Type":"ContainerStarted","Data":"1b4abd5ac1e811073f76877a80755345a7c5080e78060a1ec1c95891d48a2e02"}
Oct 30 00:34:01.317295 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:01.316865    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j" event={"ID":"2af359bf-d670-4f66-9dab-2a95b328705e","Type":"ContainerStarted","Data":"fa9894ac7c95d50fd3b68462b1900dd76a7223f09a5a731bb2a32f0c1469d7bf"}
Oct 30 00:34:01.317295 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:01.316908    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"
Oct 30 00:34:01.361544 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:01.361485    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j" podStartSLOduration=1.36146718 podStartE2EDuration="1.36146718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:01.354988048 +0000 UTC m=+1750.880281938" watchObservedRunningTime="2026-10-30 00:34:01.36146718 +0000 UTC m=+1750.886761058"
Oct 30 00:34:02.140419 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:02.140376    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:34:02.662046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:02.650950    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n" podUID="69a47c65-8652-4da3-a02d-961c57ff4d34" containerName="payload-processing" containerID="cri-o://15c3549ae465f2f755e1a21c1bfa1fad653f8334e4228b9ead9194659031c73d" gracePeriod=25
Oct 30 00:34:03.235820 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:03.235781    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n" podUID="69a47c65-8652-4da3-a02d-961c57ff4d34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:34:04.012923 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:04.012522    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp" podUID="7ffa771a-b9fc-481e-8a37-9831c31780a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:34:04.029048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:04.018177    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz" podUID="21bca81c-b78d-45a0-8c76-13b92029347e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:34:04.101049 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:34:04.099613    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:34:05.349051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:05.344102    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:34:05.428050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:05.427346    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:34:05.598051 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:05.597582    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 00:34:05.686892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:05.665238    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:05.753836 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:05.751253    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:06.045062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:06.033048    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:34:06.045062 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:06.033120    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:06.143813 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:06.142480    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:34:06.202063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:06.176679    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" containerID="cri-o://2790d95ed7afed6417752a563fd974da2076592674ac45dd4f8b0809413209ac" gracePeriod=45
Oct 30 00:34:06.202063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:06.180980    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:06.202063 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:06.181103    4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:06.437140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:06.430717    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:06.437140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:06.433356    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:34:06.800089 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:06.797152    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 00:34:06.861049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:06.852738    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:34:07.153147 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.145984    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:34:07.153147 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.146520    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:34:07.187082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.186136    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:07.187082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.186667    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:34:07.207918 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.206058    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:34:07.211229 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.210743    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:34:07.273068 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.238195    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": EOF"
Oct 30 00:34:07.335384 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.318460    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:34:07.362797 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.360494    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:34:07.696938 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.696910    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:34:07.825356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.825323    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-home\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-home\") pod \"39b25719-53ed-4967-8d2a-153ed54b739c\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") "
Oct 30 00:34:07.825551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.825375    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-tmp-dir\") pod \"39b25719-53ed-4967-8d2a-153ed54b739c\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") "
Oct 30 00:34:07.825551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.825523    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-model-cache\") pod \"39b25719-53ed-4967-8d2a-153ed54b739c\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") "
Oct 30 00:34:07.825669 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.825575    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-dshm\") pod \"39b25719-53ed-4967-8d2a-153ed54b739c\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") "
Oct 30 00:34:07.826504 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.825662    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-home" pod "39b25719-53ed-4967-8d2a-153ed54b739c" (UID: "39b25719-53ed-4967-8d2a-153ed54b739c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:07.826504 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.825703    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39b25719-53ed-4967-8d2a-153ed54b739c-tls-certs\" (UniqueName: \"kubernetes.io/secret/39b25719-53ed-4967-8d2a-153ed54b739c-tls-certs\") pod \"39b25719-53ed-4967-8d2a-153ed54b739c\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") "
Oct 30 00:34:07.826504 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.825701    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-tmp-dir" pod "39b25719-53ed-4967-8d2a-153ed54b739c" (UID: "39b25719-53ed-4967-8d2a-153ed54b739c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:07.826504 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.825749    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b25719-53ed-4967-8d2a-153ed54b739c-kube-api-access-hzwff\" (UniqueName: \"kubernetes.io/projected/39b25719-53ed-4967-8d2a-153ed54b739c-kube-api-access-hzwff\") pod \"39b25719-53ed-4967-8d2a-153ed54b739c\" (UID: \"39b25719-53ed-4967-8d2a-153ed54b739c\") "
Oct 30 00:34:07.826504 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.825800    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-model-cache" pod "39b25719-53ed-4967-8d2a-153ed54b739c" (UID: "39b25719-53ed-4967-8d2a-153ed54b739c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:07.826791 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.826665    4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-home\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:07.826791 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.826689    4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-tmp-dir\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:07.826791 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.826705    4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-model-cache\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:07.828436 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.828409    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-dshm" pod "39b25719-53ed-4967-8d2a-153ed54b739c" (UID: "39b25719-53ed-4967-8d2a-153ed54b739c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:07.828568 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.828414    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b25719-53ed-4967-8d2a-153ed54b739c-tls-certs" pod "39b25719-53ed-4967-8d2a-153ed54b739c" (UID: "39b25719-53ed-4967-8d2a-153ed54b739c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:34:07.828568 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.828522    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b25719-53ed-4967-8d2a-153ed54b739c-kube-api-access-hzwff" pod "39b25719-53ed-4967-8d2a-153ed54b739c" (UID: "39b25719-53ed-4967-8d2a-153ed54b739c"). InnerVolumeSpecName "kube-api-access-hzwff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:07.928747 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.928712    4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b25719-53ed-4967-8d2a-153ed54b739c-dshm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:07.928747 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.928749    4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b25719-53ed-4967-8d2a-153ed54b739c-tls-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:07.928953 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:07.928764    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzwff\" (UniqueName: \"kubernetes.io/projected/39b25719-53ed-4967-8d2a-153ed54b739c-kube-api-access-hzwff\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:08.406228 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:08.405750    4441 generic.go:356] "Generic (PLEG): container finished" podID="39b25719-53ed-4967-8d2a-153ed54b739c" containerID="2790d95ed7afed6417752a563fd974da2076592674ac45dd4f8b0809413209ac" exitCode=0
Oct 30 00:34:08.407884 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:08.407684    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"
Oct 30 00:34:08.413156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:08.413122    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" event={"ID":"39b25719-53ed-4967-8d2a-153ed54b739c","Type":"ContainerDied","Data":"2790d95ed7afed6417752a563fd974da2076592674ac45dd4f8b0809413209ac"}
Oct 30 00:34:08.413265 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:08.413179    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" event={"ID":"39b25719-53ed-4967-8d2a-153ed54b739c","Type":"ContainerDied","Data":"c1547a61a125aab2b005ca55dc31db4c38000872f6dc4d5ff22911ea56bf1d53"}
Oct 30 00:34:08.413265 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:08.413212    4441 scope.go:122] "RemoveContainer" containerID="2790d95ed7afed6417752a563fd974da2076592674ac45dd4f8b0809413209ac"
Oct 30 00:34:08.477048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:08.476270    4441 scope.go:122] "RemoveContainer" containerID="2790d95ed7afed6417752a563fd974da2076592674ac45dd4f8b0809413209ac"
Oct 30 00:34:08.477453 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:34:08.477424    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2790d95ed7afed6417752a563fd974da2076592674ac45dd4f8b0809413209ac\": container with ID starting with 2790d95ed7afed6417752a563fd974da2076592674ac45dd4f8b0809413209ac not found: ID does not exist" containerID="2790d95ed7afed6417752a563fd974da2076592674ac45dd4f8b0809413209ac"
Oct 30 00:34:08.477636 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:08.477610    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2790d95ed7afed6417752a563fd974da2076592674ac45dd4f8b0809413209ac"} err="failed to get container status \"2790d95ed7afed6417752a563fd974da2076592674ac45dd4f8b0809413209ac\": rpc error: code = NotFound desc = could not find container \"2790d95ed7afed6417752a563fd974da2076592674ac45dd4f8b0809413209ac\": container with ID starting with 2790d95ed7afed6417752a563fd974da2076592674ac45dd4f8b0809413209ac not found: ID does not exist"
Oct 30 00:34:08.518902 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:08.518865    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"]
Oct 30 00:34:08.533084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:08.529846    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn"]
Oct 30 00:34:08.611979 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:08.611941    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" path="/var/lib/kubelet/pods/39b25719-53ed-4967-8d2a-153ed54b739c/volumes"
Oct 30 00:34:08.646221 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:08.646168    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/premium-w3-564c8f-kserve-57b794fcb7-j2hwn" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:12.353407 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:12.353381    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"
Oct 30 00:34:13.237489 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:13.237452    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n" podUID="69a47c65-8652-4da3-a02d-961c57ff4d34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:34:13.990661 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:13.990624    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp" podUID="7ffa771a-b9fc-481e-8a37-9831c31780a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:34:13.991478 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:13.991450    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz" podUID="21bca81c-b78d-45a0-8c76-13b92029347e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:34:16.494927 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.494897    4441 generic.go:356] "Generic (PLEG): container finished" podID="7ffa771a-b9fc-481e-8a37-9831c31780a2" containerID="3d9167bd5a3d7dea9a28bda44846fa7d439da6b97d199c25655432f139856b07" exitCode=0
Oct 30 00:34:16.495347 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.494947    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp" event={"ID":"7ffa771a-b9fc-481e-8a37-9831c31780a2","Type":"ContainerDied","Data":"3d9167bd5a3d7dea9a28bda44846fa7d439da6b97d199c25655432f139856b07"}
Oct 30 00:34:16.496416 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.496391    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz" event={"ID":"21bca81c-b78d-45a0-8c76-13b92029347e","Type":"ContainerDied","Data":"6996d4f1ff17ccc6899a9e762d0804c606cc548c0fc937cbca2ef2620378ef1d"}
Oct 30 00:34:16.496416 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.496318    4441 generic.go:356] "Generic (PLEG): container finished" podID="21bca81c-b78d-45a0-8c76-13b92029347e" containerID="6996d4f1ff17ccc6899a9e762d0804c606cc548c0fc937cbca2ef2620378ef1d" exitCode=0
Oct 30 00:34:16.496557 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.496441    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz" event={"ID":"21bca81c-b78d-45a0-8c76-13b92029347e","Type":"ContainerDied","Data":"a8f1f238dfc49e151b1c86b0be9e3d7b8cb67bfbcb036133307d0878b8a5d68e"}
Oct 30 00:34:16.496557 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.496464    4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8f1f238dfc49e151b1c86b0be9e3d7b8cb67bfbcb036133307d0878b8a5d68e"
Oct 30 00:34:16.496557 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.496521    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"
Oct 30 00:34:16.520391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.520371    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21bca81c-b78d-45a0-8c76-13b92029347e-kube-api-access-nb922\" (UniqueName: \"kubernetes.io/projected/21bca81c-b78d-45a0-8c76-13b92029347e-kube-api-access-nb922\") pod \"21bca81c-b78d-45a0-8c76-13b92029347e\" (UID: \"21bca81c-b78d-45a0-8c76-13b92029347e\") "
Oct 30 00:34:16.520491 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.520426    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21bca81c-b78d-45a0-8c76-13b92029347e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bca81c-b78d-45a0-8c76-13b92029347e-plugins-config-volume\") pod \"21bca81c-b78d-45a0-8c76-13b92029347e\" (UID: \"21bca81c-b78d-45a0-8c76-13b92029347e\") "
Oct 30 00:34:16.520761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.520741    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21bca81c-b78d-45a0-8c76-13b92029347e-plugins-config-volume" pod "21bca81c-b78d-45a0-8c76-13b92029347e" (UID: "21bca81c-b78d-45a0-8c76-13b92029347e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:34:16.524074 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.521011    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bca81c-b78d-45a0-8c76-13b92029347e-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:16.524074 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.522672    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bca81c-b78d-45a0-8c76-13b92029347e-kube-api-access-nb922" pod "21bca81c-b78d-45a0-8c76-13b92029347e" (UID: "21bca81c-b78d-45a0-8c76-13b92029347e"). InnerVolumeSpecName "kube-api-access-nb922". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:16.525843 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.525826    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"
Oct 30 00:34:16.621423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.621393    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ffa771a-b9fc-481e-8a37-9831c31780a2-kube-api-access-hqlgr\" (UniqueName: \"kubernetes.io/projected/7ffa771a-b9fc-481e-8a37-9831c31780a2-kube-api-access-hqlgr\") pod \"7ffa771a-b9fc-481e-8a37-9831c31780a2\" (UID: \"7ffa771a-b9fc-481e-8a37-9831c31780a2\") "
Oct 30 00:34:16.621614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.621563    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ffa771a-b9fc-481e-8a37-9831c31780a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ffa771a-b9fc-481e-8a37-9831c31780a2-plugins-config-volume\") pod \"7ffa771a-b9fc-481e-8a37-9831c31780a2\" (UID: \"7ffa771a-b9fc-481e-8a37-9831c31780a2\") "
Oct 30 00:34:16.622312 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.622263    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ffa771a-b9fc-481e-8a37-9831c31780a2-plugins-config-volume" pod "7ffa771a-b9fc-481e-8a37-9831c31780a2" (UID: "7ffa771a-b9fc-481e-8a37-9831c31780a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:34:16.622822 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.622771    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb922\" (UniqueName: \"kubernetes.io/projected/21bca81c-b78d-45a0-8c76-13b92029347e-kube-api-access-nb922\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:16.622822 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.622798    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ffa771a-b9fc-481e-8a37-9831c31780a2-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:16.625295 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.625272    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ffa771a-b9fc-481e-8a37-9831c31780a2-kube-api-access-hqlgr" pod "7ffa771a-b9fc-481e-8a37-9831c31780a2" (UID: "7ffa771a-b9fc-481e-8a37-9831c31780a2"). InnerVolumeSpecName "kube-api-access-hqlgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:16.724712 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:16.724646    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqlgr\" (UniqueName: \"kubernetes.io/projected/7ffa771a-b9fc-481e-8a37-9831c31780a2-kube-api-access-hqlgr\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:17.504713 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:17.504677    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp" event={"ID":"7ffa771a-b9fc-481e-8a37-9831c31780a2","Type":"ContainerDied","Data":"80801fbfbdd0f4fdec841ce4f65ce7a3f63a30c51931a0ef178dcb03d826741c"}
Oct 30 00:34:17.505234 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:17.504742    4441 scope.go:122] "RemoveContainer" containerID="3d9167bd5a3d7dea9a28bda44846fa7d439da6b97d199c25655432f139856b07"
Oct 30 00:34:17.505234 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:17.504685    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"
Oct 30 00:34:17.505234 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:17.504784    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"
Oct 30 00:34:17.573272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:17.573242    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"]
Oct 30 00:34:17.583013 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:17.582985    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b1c718-7c9cf4fdc5-gh8vp"]
Oct 30 00:34:17.606610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:17.606580    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"]
Oct 30 00:34:17.610483 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:17.610451    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b1c718-55545565b8-5zfcz"]
Oct 30 00:34:18.610426 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:18.610397    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21bca81c-b78d-45a0-8c76-13b92029347e" path="/var/lib/kubelet/pods/21bca81c-b78d-45a0-8c76-13b92029347e/volumes"
Oct 30 00:34:18.610849 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:18.610834    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ffa771a-b9fc-481e-8a37-9831c31780a2" path="/var/lib/kubelet/pods/7ffa771a-b9fc-481e-8a37-9831c31780a2/volumes"
Oct 30 00:34:21.148749 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.148713    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"]
Oct 30 00:34:21.149509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.149492    4441 cpu_manager.go:414] "containerMap: removing container" podUID="21bca81c-b78d-45a0-8c76-13b92029347e" containerName="payload-processing"
Oct 30 00:34:21.149555 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.149511    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21bca81c-b78d-45a0-8c76-13b92029347e" containerName="payload-processing"
Oct 30 00:34:21.149555 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.149521    4441 cpu_manager.go:414] "containerMap: removing container" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main"
Oct 30 00:34:21.149555 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.149526    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main"
Oct 30 00:34:21.149555 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.149539    4441 cpu_manager.go:414] "containerMap: removing container" podUID="7ffa771a-b9fc-481e-8a37-9831c31780a2" containerName="payload-pre-processing"
Oct 30 00:34:21.149555 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.149545    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ffa771a-b9fc-481e-8a37-9831c31780a2" containerName="payload-pre-processing"
Oct 30 00:34:21.149713 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.149645    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="21bca81c-b78d-45a0-8c76-13b92029347e" containerName="payload-processing"
Oct 30 00:34:21.149713 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.149654    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b25719-53ed-4967-8d2a-153ed54b739c" containerName="main"
Oct 30 00:34:21.149713 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.149662    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ffa771a-b9fc-481e-8a37-9831c31780a2" containerName="payload-pre-processing"
Oct 30 00:34:21.154094 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.154067    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.170629 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.170601    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-f1268c-serving-cert\""
Oct 30 00:34:21.184609 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.184582    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"]
Oct 30 00:34:21.201599 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.201570    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.201755 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.201735    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.201879 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.201861    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b439da36-a99a-4847-9884-1c19d9d0b8e8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.202009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.201990    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcxcz\" (UniqueName: \"kubernetes.io/projected/b439da36-a99a-4847-9884-1c19d9d0b8e8-kube-api-access-gcxcz\") pod \"maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.303528 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.303496    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.303692 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.303573    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.303692 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.303627    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b439da36-a99a-4847-9884-1c19d9d0b8e8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.303692 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.303679    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcxcz\" (UniqueName: \"kubernetes.io/projected/b439da36-a99a-4847-9884-1c19d9d0b8e8-kube-api-access-gcxcz\") pod \"maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.304345 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.304317    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b439da36-a99a-4847-9884-1c19d9d0b8e8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.306095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.306051    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.306095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.306087    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.326238 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.326202    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcxcz\" (UniqueName: \"kubernetes.io/projected/b439da36-a99a-4847-9884-1c19d9d0b8e8-kube-api-access-gcxcz\") pod \"maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.471677 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.471583    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:21.819833 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:21.819806    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"]
Oct 30 00:34:21.820180 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:34:21.820151    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb439da36_a99a_4847_9884_1c19d9d0b8e8.slice/crio-bf774bea881ed1e7be17253233dccb81b37d36ca3bd7768522dfb86877696bf1 WatchSource:0}: Error finding container bf774bea881ed1e7be17253233dccb81b37d36ca3bd7768522dfb86877696bf1: Status 404 returned error can't find the container with id bf774bea881ed1e7be17253233dccb81b37d36ca3bd7768522dfb86877696bf1
Oct 30 00:34:22.575457 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:22.575423    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5" event={"ID":"b439da36-a99a-4847-9884-1c19d9d0b8e8","Type":"ContainerStarted","Data":"6e58a3f112470779ee285b4d1f3857674b91f5a74285967f6e8156720f5e3c47"}
Oct 30 00:34:22.575938 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:22.575473    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5" event={"ID":"b439da36-a99a-4847-9884-1c19d9d0b8e8","Type":"ContainerStarted","Data":"bf774bea881ed1e7be17253233dccb81b37d36ca3bd7768522dfb86877696bf1"}
Oct 30 00:34:22.575938 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:22.575557    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:22.625050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:22.624976    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5" podStartSLOduration=1.275988207 podStartE2EDuration="1.62495576s" totalImagesPullingTime="348.967553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:22.620240498 +0000 UTC m=+1772.145534376" watchObservedRunningTime="2026-10-30 00:34:22.62495576 +0000 UTC m=+1772.150249638"
Oct 30 00:34:23.035737 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.035704    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"
Oct 30 00:34:23.165981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.165908    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a47c65-8652-4da3-a02d-961c57ff4d34-kube-api-access-fq6jv\" (UniqueName: \"kubernetes.io/projected/69a47c65-8652-4da3-a02d-961c57ff4d34-kube-api-access-fq6jv\") pod \"69a47c65-8652-4da3-a02d-961c57ff4d34\" (UID: \"69a47c65-8652-4da3-a02d-961c57ff4d34\") "
Oct 30 00:34:23.166206 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.166191    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a47c65-8652-4da3-a02d-961c57ff4d34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a47c65-8652-4da3-a02d-961c57ff4d34-plugins-config-volume\") pod \"69a47c65-8652-4da3-a02d-961c57ff4d34\" (UID: \"69a47c65-8652-4da3-a02d-961c57ff4d34\") "
Oct 30 00:34:23.168161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.166933    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a47c65-8652-4da3-a02d-961c57ff4d34-plugins-config-volume" pod "69a47c65-8652-4da3-a02d-961c57ff4d34" (UID: "69a47c65-8652-4da3-a02d-961c57ff4d34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:34:23.168161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.167809    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a47c65-8652-4da3-a02d-961c57ff4d34-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:23.181407 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.181371    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a47c65-8652-4da3-a02d-961c57ff4d34-kube-api-access-fq6jv" pod "69a47c65-8652-4da3-a02d-961c57ff4d34" (UID: "69a47c65-8652-4da3-a02d-961c57ff4d34"). InnerVolumeSpecName "kube-api-access-fq6jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:23.269841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.269802    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq6jv\" (UniqueName: \"kubernetes.io/projected/69a47c65-8652-4da3-a02d-961c57ff4d34-kube-api-access-fq6jv\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:23.583828 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.583795    4441 generic.go:356] "Generic (PLEG): container finished" podID="69a47c65-8652-4da3-a02d-961c57ff4d34" containerID="15c3549ae465f2f755e1a21c1bfa1fad653f8334e4228b9ead9194659031c73d" exitCode=0
Oct 30 00:34:23.584344 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.583854    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"
Oct 30 00:34:23.584344 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.583881    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n" event={"ID":"69a47c65-8652-4da3-a02d-961c57ff4d34","Type":"ContainerDied","Data":"15c3549ae465f2f755e1a21c1bfa1fad653f8334e4228b9ead9194659031c73d"}
Oct 30 00:34:23.584344 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.583982    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n" event={"ID":"69a47c65-8652-4da3-a02d-961c57ff4d34","Type":"ContainerDied","Data":"9dce98c63724e4a0a0556f72686676b60de4128d3120609e6cdbce30dd0d5a47"}
Oct 30 00:34:23.584344 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.584015    4441 scope.go:122] "RemoveContainer" containerID="15c3549ae465f2f755e1a21c1bfa1fad653f8334e4228b9ead9194659031c73d"
Oct 30 00:34:23.595891 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.595871    4441 scope.go:122] "RemoveContainer" containerID="15c3549ae465f2f755e1a21c1bfa1fad653f8334e4228b9ead9194659031c73d"
Oct 30 00:34:23.596185 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:34:23.596163    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c3549ae465f2f755e1a21c1bfa1fad653f8334e4228b9ead9194659031c73d\": container with ID starting with 15c3549ae465f2f755e1a21c1bfa1fad653f8334e4228b9ead9194659031c73d not found: ID does not exist" containerID="15c3549ae465f2f755e1a21c1bfa1fad653f8334e4228b9ead9194659031c73d"
Oct 30 00:34:23.596279 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.596189    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c3549ae465f2f755e1a21c1bfa1fad653f8334e4228b9ead9194659031c73d"} err="failed to get container status \"15c3549ae465f2f755e1a21c1bfa1fad653f8334e4228b9ead9194659031c73d\": rpc error: code = NotFound desc = could not find container \"15c3549ae465f2f755e1a21c1bfa1fad653f8334e4228b9ead9194659031c73d\": container with ID starting with 15c3549ae465f2f755e1a21c1bfa1fad653f8334e4228b9ead9194659031c73d not found: ID does not exist"
Oct 30 00:34:23.636328 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.636300    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"]
Oct 30 00:34:23.647391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:23.647363    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-564c8f-5558b4d9cb-c6b2n"]
Oct 30 00:34:24.614325 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:24.614288    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a47c65-8652-4da3-a02d-961c57ff4d34" path="/var/lib/kubelet/pods/69a47c65-8652-4da3-a02d-961c57ff4d34/volumes"
Oct 30 00:34:28.924050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:28.923924    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:34:29.214224 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:29.214116    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:29.334708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:29.214629    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:29.334708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:29.324189    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:34:29.421558 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:29.421517    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 00:34:29.563109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:29.563064    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:29.577531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:29.577382    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:30.305515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:30.303572    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:30.307114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:30.306321    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:34:30.421101 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:30.416551    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:34:30.438799 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:30.438165    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:30.441737 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:30.439237    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:34:30.818411 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:30.818383    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:34:32.472141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:32.471200    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"]
Oct 30 00:34:32.472141 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:32.471488    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy" containerID="cri-o://bd46f99662b1cd397cce54760b890fbab9b0fedb7fe86f1a7264e89df240f774" gracePeriod=30
Oct 30 00:34:32.515720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:32.514444    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"]
Oct 30 00:34:32.515720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:32.514703    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" containerName="istio-proxy" containerID="cri-o://a46bb8f9449d3200c8f1aacf9be7a70a7e6915622eb2b78642d0f917b58c2ab2" gracePeriod=30
Oct 30 00:34:32.533061 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:32.520386    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"]
Oct 30 00:34:32.575157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:32.573521    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerName="istio-proxy" containerID="cri-o://94ae8fff20855af9f41573af92274f8ac6f70c94dcdbf230b9dfe37a2288daf4" gracePeriod=30
Oct 30 00:34:33.990077 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:34:33.990010    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:34:34.965240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:34.965195    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:34:34.965438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:34.965265    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:34:35.640491 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:35.640160    4441 patch_prober.go:29] interesting pod/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:34:35.640491 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:35.640229    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:34:37.429959 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:37.429917    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:34:37.430493 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:37.429986    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:34:37.966740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:37.965755    4441 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:34:37.966740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:37.965822    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:34:38.066436 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:38.066395    4441 generic.go:356] "Generic (PLEG): container finished" podID="cc27936a-64b1-473d-b857-07abf44fdacf" containerID="a46bb8f9449d3200c8f1aacf9be7a70a7e6915622eb2b78642d0f917b58c2ab2" exitCode=0
Oct 30 00:34:38.066610 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:38.066533    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" event={"ID":"cc27936a-64b1-473d-b857-07abf44fdacf","Type":"ContainerDied","Data":"a46bb8f9449d3200c8f1aacf9be7a70a7e6915622eb2b78642d0f917b58c2ab2"}
Oct 30 00:34:38.069242 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:38.069209    4441 generic.go:356] "Generic (PLEG): container finished" podID="daae17d4-1319-4ef7-adb7-a013358915db" containerID="bd46f99662b1cd397cce54760b890fbab9b0fedb7fe86f1a7264e89df240f774" exitCode=0
Oct 30 00:34:38.069374 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:38.069293    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" event={"ID":"daae17d4-1319-4ef7-adb7-a013358915db","Type":"ContainerDied","Data":"bd46f99662b1cd397cce54760b890fbab9b0fedb7fe86f1a7264e89df240f774"}
Oct 30 00:34:38.069374 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:38.069338    4441 scope.go:122] "RemoveContainer" containerID="9b02c56d3aad4dd1c1ec9d2b66eed5a6acf66073ad5f732b3a4452bde113c547"
Oct 30 00:34:38.074585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:38.073727    4441 generic.go:356] "Generic (PLEG): container finished" podID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerID="94ae8fff20855af9f41573af92274f8ac6f70c94dcdbf230b9dfe37a2288daf4" exitCode=0
Oct 30 00:34:38.074585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:38.073856    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" event={"ID":"d355c2f5-010e-409c-81e7-8f1ff07db365","Type":"ContainerDied","Data":"94ae8fff20855af9f41573af92274f8ac6f70c94dcdbf230b9dfe37a2288daf4"}
Oct 30 00:34:39.040536 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.040073    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:34:39.088175 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.087626    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd" event={"ID":"daae17d4-1319-4ef7-adb7-a013358915db","Type":"ContainerDied","Data":"4710b885863f962d4cc3e20be947299ae057e12c9122998108ac463e664e6e3d"}
Oct 30 00:34:39.088175 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.087701    4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4710b885863f962d4cc3e20be947299ae057e12c9122998108ac463e664e6e3d"
Oct 30 00:34:39.090337 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.090304    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c" event={"ID":"d355c2f5-010e-409c-81e7-8f1ff07db365","Type":"ContainerDied","Data":"c4c90fa45308973ba2ca475ad09527ea63824a69b2618085b90d2ea80ad5d0ad"}
Oct 30 00:34:39.090459 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.090364    4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4c90fa45308973ba2ca475ad09527ea63824a69b2618085b90d2ea80ad5d0ad"
Oct 30 00:34:39.092467 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.092441    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk" event={"ID":"cc27936a-64b1-473d-b857-07abf44fdacf","Type":"ContainerDied","Data":"70b4584c44f485d9740bcf6a6320b6151a43672e1cb293273e401b1f328726ed"}
Oct 30 00:34:39.092615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.092506    4441 scope.go:122] "RemoveContainer" containerID="a46bb8f9449d3200c8f1aacf9be7a70a7e6915622eb2b78642d0f917b58c2ab2"
Oct 30 00:34:39.092615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.092451    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"
Oct 30 00:34:39.171711 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.171682    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-istio-token\" (UniqueName: \"kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-istio-token\") pod \"cc27936a-64b1-473d-b857-07abf44fdacf\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") "
Oct 30 00:34:39.171890 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.171831    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-envoy\") pod \"cc27936a-64b1-473d-b857-07abf44fdacf\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") "
Oct 30 00:34:39.171890 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.171864    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-socket\") pod \"cc27936a-64b1-473d-b857-07abf44fdacf\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") "
Oct 30 00:34:39.172014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.171982    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc27936a-64b1-473d-b857-07abf44fdacf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc27936a-64b1-473d-b857-07abf44fdacf-istiod-ca-cert\") pod \"cc27936a-64b1-473d-b857-07abf44fdacf\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") "
Oct 30 00:34:39.172117 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.172084    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-data\") pod \"cc27936a-64b1-473d-b857-07abf44fdacf\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") "
Oct 30 00:34:39.172179 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.172159    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc27936a-64b1-473d-b857-07abf44fdacf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc27936a-64b1-473d-b857-07abf44fdacf-istio-podinfo\") pod \"cc27936a-64b1-473d-b857-07abf44fdacf\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") "
Oct 30 00:34:39.172240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.172198    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-credential-socket\") pod \"cc27936a-64b1-473d-b857-07abf44fdacf\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") "
Oct 30 00:34:39.172300 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.172239    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-kube-api-access-wfqrh\" (UniqueName: \"kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-kube-api-access-wfqrh\") pod \"cc27936a-64b1-473d-b857-07abf44fdacf\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") "
Oct 30 00:34:39.172300 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.172273    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-certs\") pod \"cc27936a-64b1-473d-b857-07abf44fdacf\" (UID: \"cc27936a-64b1-473d-b857-07abf44fdacf\") "
Oct 30 00:34:39.172492 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.172462    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc27936a-64b1-473d-b857-07abf44fdacf-istiod-ca-cert" pod "cc27936a-64b1-473d-b857-07abf44fdacf" (UID: "cc27936a-64b1-473d-b857-07abf44fdacf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:34:39.173351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.172903    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-socket" pod "cc27936a-64b1-473d-b857-07abf44fdacf" (UID: "cc27936a-64b1-473d-b857-07abf44fdacf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.173351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.172997    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-certs" pod "cc27936a-64b1-473d-b857-07abf44fdacf" (UID: "cc27936a-64b1-473d-b857-07abf44fdacf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.173351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.173287    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.173351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.173312    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc27936a-64b1-473d-b857-07abf44fdacf-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.173351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.173329    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.173784 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.173746    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-data" pod "cc27936a-64b1-473d-b857-07abf44fdacf" (UID: "cc27936a-64b1-473d-b857-07abf44fdacf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.173928 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.173909    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-credential-socket" pod "cc27936a-64b1-473d-b857-07abf44fdacf" (UID: "cc27936a-64b1-473d-b857-07abf44fdacf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.174532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.174505    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-envoy" pod "cc27936a-64b1-473d-b857-07abf44fdacf" (UID: "cc27936a-64b1-473d-b857-07abf44fdacf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.174612 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.174534    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-istio-token" pod "cc27936a-64b1-473d-b857-07abf44fdacf" (UID: "cc27936a-64b1-473d-b857-07abf44fdacf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:39.174749 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.174724    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc27936a-64b1-473d-b857-07abf44fdacf-istio-podinfo" pod "cc27936a-64b1-473d-b857-07abf44fdacf" (UID: "cc27936a-64b1-473d-b857-07abf44fdacf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:34:39.175103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.175084    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-kube-api-access-wfqrh" pod "cc27936a-64b1-473d-b857-07abf44fdacf" (UID: "cc27936a-64b1-473d-b857-07abf44fdacf"). InnerVolumeSpecName "kube-api-access-wfqrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:39.237658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.237629    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:34:39.275259 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.275228    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.275259 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.275258    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc27936a-64b1-473d-b857-07abf44fdacf-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.275428 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.275272    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.275428 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.275287    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfqrh\" (UniqueName: \"kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-kube-api-access-wfqrh\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.275428 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.275310    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc27936a-64b1-473d-b857-07abf44fdacf-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.275428 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.275322    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc27936a-64b1-473d-b857-07abf44fdacf-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.335008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.334945    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:34:39.383376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.382966    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-envoy\") pod \"daae17d4-1319-4ef7-adb7-a013358915db\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") "
Oct 30 00:34:39.383376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.383046    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-kube-api-access-zpkch\" (UniqueName: \"kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-kube-api-access-zpkch\") pod \"daae17d4-1319-4ef7-adb7-a013358915db\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") "
Oct 30 00:34:39.383376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.383148    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/daae17d4-1319-4ef7-adb7-a013358915db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daae17d4-1319-4ef7-adb7-a013358915db-istio-podinfo\") pod \"daae17d4-1319-4ef7-adb7-a013358915db\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") "
Oct 30 00:34:39.383376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.383194    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-credential-socket\") pod \"daae17d4-1319-4ef7-adb7-a013358915db\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") "
Oct 30 00:34:39.383376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.383229    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-socket\") pod \"daae17d4-1319-4ef7-adb7-a013358915db\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") "
Oct 30 00:34:39.383376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.383297    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daae17d4-1319-4ef7-adb7-a013358915db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daae17d4-1319-4ef7-adb7-a013358915db-istiod-ca-cert\") pod \"daae17d4-1319-4ef7-adb7-a013358915db\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") "
Oct 30 00:34:39.383813 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.383419    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-certs\") pod \"daae17d4-1319-4ef7-adb7-a013358915db\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") "
Oct 30 00:34:39.383813 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.383579    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-istio-token\" (UniqueName: \"kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-istio-token\") pod \"daae17d4-1319-4ef7-adb7-a013358915db\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") "
Oct 30 00:34:39.383813 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.383624    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-data\") pod \"daae17d4-1319-4ef7-adb7-a013358915db\" (UID: \"daae17d4-1319-4ef7-adb7-a013358915db\") "
Oct 30 00:34:39.384710 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.384687    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-credential-socket" pod "daae17d4-1319-4ef7-adb7-a013358915db" (UID: "daae17d4-1319-4ef7-adb7-a013358915db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.384867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.384695    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-data" pod "daae17d4-1319-4ef7-adb7-a013358915db" (UID: "daae17d4-1319-4ef7-adb7-a013358915db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.384980 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.384900    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-socket" pod "daae17d4-1319-4ef7-adb7-a013358915db" (UID: "daae17d4-1319-4ef7-adb7-a013358915db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.385226 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.385205    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daae17d4-1319-4ef7-adb7-a013358915db-istiod-ca-cert" pod "daae17d4-1319-4ef7-adb7-a013358915db" (UID: "daae17d4-1319-4ef7-adb7-a013358915db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:34:39.385482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.385454    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-certs" pod "daae17d4-1319-4ef7-adb7-a013358915db" (UID: "daae17d4-1319-4ef7-adb7-a013358915db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.389655 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.388015    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-istio-token" pod "daae17d4-1319-4ef7-adb7-a013358915db" (UID: "daae17d4-1319-4ef7-adb7-a013358915db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:39.389655 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.389583    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-kube-api-access-zpkch" pod "daae17d4-1319-4ef7-adb7-a013358915db" (UID: "daae17d4-1319-4ef7-adb7-a013358915db"). InnerVolumeSpecName "kube-api-access-zpkch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:39.391494 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.391468    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/daae17d4-1319-4ef7-adb7-a013358915db-istio-podinfo" pod "daae17d4-1319-4ef7-adb7-a013358915db" (UID: "daae17d4-1319-4ef7-adb7-a013358915db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:34:39.392161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.392134    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-envoy" pod "daae17d4-1319-4ef7-adb7-a013358915db" (UID: "daae17d4-1319-4ef7-adb7-a013358915db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.485822 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.485788    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d355c2f5-010e-409c-81e7-8f1ff07db365-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d355c2f5-010e-409c-81e7-8f1ff07db365-istiod-ca-cert\") pod \"d355c2f5-010e-409c-81e7-8f1ff07db365\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") "
Oct 30 00:34:39.485985 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.485881    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-kube-api-access-7j86q\" (UniqueName: \"kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-kube-api-access-7j86q\") pod \"d355c2f5-010e-409c-81e7-8f1ff07db365\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") "
Oct 30 00:34:39.485985 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.485903    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-data\") pod \"d355c2f5-010e-409c-81e7-8f1ff07db365\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") "
Oct 30 00:34:39.485985 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.485930    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-credential-socket\") pod \"d355c2f5-010e-409c-81e7-8f1ff07db365\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") "
Oct 30 00:34:39.485985 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.485960    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-envoy\") pod \"d355c2f5-010e-409c-81e7-8f1ff07db365\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") "
Oct 30 00:34:39.486186 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.485994    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-token\" (UniqueName: \"kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-token\") pod \"d355c2f5-010e-409c-81e7-8f1ff07db365\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") "
Oct 30 00:34:39.486186 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.486099    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-podinfo\") pod \"d355c2f5-010e-409c-81e7-8f1ff07db365\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") "
Oct 30 00:34:39.486186 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.486169    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-certs\") pod \"d355c2f5-010e-409c-81e7-8f1ff07db365\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") "
Oct 30 00:34:39.486314 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.486211    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-socket\") pod \"d355c2f5-010e-409c-81e7-8f1ff07db365\" (UID: \"d355c2f5-010e-409c-81e7-8f1ff07db365\") "
Oct 30 00:34:39.486670 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.486647    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-credential-socket" pod "d355c2f5-010e-409c-81e7-8f1ff07db365" (UID: "d355c2f5-010e-409c-81e7-8f1ff07db365"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.487124 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.487097    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-data" pod "d355c2f5-010e-409c-81e7-8f1ff07db365" (UID: "d355c2f5-010e-409c-81e7-8f1ff07db365"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.487343 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.487114    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-certs" pod "d355c2f5-010e-409c-81e7-8f1ff07db365" (UID: "d355c2f5-010e-409c-81e7-8f1ff07db365"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.487496 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.487426    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d355c2f5-010e-409c-81e7-8f1ff07db365-istiod-ca-cert" pod "d355c2f5-010e-409c-81e7-8f1ff07db365" (UID: "d355c2f5-010e-409c-81e7-8f1ff07db365"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:34:39.488278 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.488221    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.488429 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.488413    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.488522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.488510    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.488602 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.488592    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.488805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.488688    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.488805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.488703    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.488805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.488715    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpkch\" (UniqueName: \"kubernetes.io/projected/daae17d4-1319-4ef7-adb7-a013358915db-kube-api-access-zpkch\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.488805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.488728    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daae17d4-1319-4ef7-adb7-a013358915db-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.488805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.488740    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.488805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.488752    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.488805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.488764    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daae17d4-1319-4ef7-adb7-a013358915db-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.488805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.488776    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d355c2f5-010e-409c-81e7-8f1ff07db365-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.488805 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.488787    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daae17d4-1319-4ef7-adb7-a013358915db-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.489398 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.489062    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-socket" pod "d355c2f5-010e-409c-81e7-8f1ff07db365" (UID: "d355c2f5-010e-409c-81e7-8f1ff07db365"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.489573 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.489540    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-podinfo" pod "d355c2f5-010e-409c-81e7-8f1ff07db365" (UID: "d355c2f5-010e-409c-81e7-8f1ff07db365"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:34:39.489761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.489745    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-token" pod "d355c2f5-010e-409c-81e7-8f1ff07db365" (UID: "d355c2f5-010e-409c-81e7-8f1ff07db365"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:39.491516 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.491494    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-kube-api-access-7j86q" pod "d355c2f5-010e-409c-81e7-8f1ff07db365" (UID: "d355c2f5-010e-409c-81e7-8f1ff07db365"). InnerVolumeSpecName "kube-api-access-7j86q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:39.491829 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.491805    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-envoy" pod "d355c2f5-010e-409c-81e7-8f1ff07db365" (UID: "d355c2f5-010e-409c-81e7-8f1ff07db365"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:39.561443 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.560298    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"]
Oct 30 00:34:39.568263 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.568214    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-6xftk"]
Oct 30 00:34:39.590397 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.590365    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j86q\" (UniqueName: \"kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-kube-api-access-7j86q\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.590611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.590401    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.590611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.590418    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.590611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.590433    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d355c2f5-010e-409c-81e7-8f1ff07db365-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:39.590611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:39.590445    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d355c2f5-010e-409c-81e7-8f1ff07db365-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:40.101750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:40.101671    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"
Oct 30 00:34:40.102275 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:40.101675    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"
Oct 30 00:34:40.169240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:40.169154    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"]
Oct 30 00:34:40.174034 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:40.173996    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-4g7fd"]
Oct 30 00:34:40.200771 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:40.200738    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"]
Oct 30 00:34:40.205371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:40.205350    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-bsh7c"]
Oct 30 00:34:40.652421 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:40.652358    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" path="/var/lib/kubelet/pods/cc27936a-64b1-473d-b857-07abf44fdacf/volumes"
Oct 30 00:34:40.669482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:40.669439    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" path="/var/lib/kubelet/pods/d355c2f5-010e-409c-81e7-8f1ff07db365/volumes"
Oct 30 00:34:40.677835 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:40.670357    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daae17d4-1319-4ef7-adb7-a013358915db" path="/var/lib/kubelet/pods/daae17d4-1319-4ef7-adb7-a013358915db/volumes"
Oct 30 00:34:42.117115 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:42.117017    4441 generic.go:356] "Generic (PLEG): container finished" podID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerID="ed24d7c957fc8094b9eec347e8b9e6bedb792ad631904b3647b80e6034f41684" exitCode=0
Oct 30 00:34:42.117115 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:42.117089    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" event={"ID":"b66cdbd4-e6a1-44e2-b459-c1502f57225a","Type":"ContainerDied","Data":"ed24d7c957fc8094b9eec347e8b9e6bedb792ad631904b3647b80e6034f41684"}
Oct 30 00:34:42.117637 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:42.117621    4441 scope.go:122] "RemoveContainer" containerID="ed24d7c957fc8094b9eec347e8b9e6bedb792ad631904b3647b80e6034f41684"
Oct 30 00:34:42.118939 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:42.118919    4441 generic.go:356] "Generic (PLEG): container finished" podID="5c0fec19-a16c-40a1-8840-66022d000e75" containerID="227177ae1363bdfb5b2019cfc6fc273f4814a4fa1a8ab60ed83cc318a187b931" exitCode=0
Oct 30 00:34:42.119080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:42.118991    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" event={"ID":"5c0fec19-a16c-40a1-8840-66022d000e75","Type":"ContainerDied","Data":"227177ae1363bdfb5b2019cfc6fc273f4814a4fa1a8ab60ed83cc318a187b931"}
Oct 30 00:34:42.119379 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:42.119365    4441 scope.go:122] "RemoveContainer" containerID="227177ae1363bdfb5b2019cfc6fc273f4814a4fa1a8ab60ed83cc318a187b931"
Oct 30 00:34:42.119438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:42.119414    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:34:42.119493 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:42.119481    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:34:42.119525 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:42.119514    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:34:42.120956 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:42.120928    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:34:42.121052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:42.120989    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:34:42.121103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:42.121020    4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:34:43.105492 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.105445    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"]
Oct 30 00:34:43.106833 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.106787    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth" podUID="4a713f91-8d61-4d25-84d1-8121a511bd94" containerName="istio-proxy" containerID="cri-o://1c31a27cb70a0863a5477e9973186b0540b45fcb2369d9996d6a405970730fdc" gracePeriod=30
Oct 30 00:34:43.116556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.116510    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"]
Oct 30 00:34:43.116964 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.116923    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c" podUID="e9347cdc-b854-4c2f-b056-e01dd2e24b2c" containerName="istio-proxy" containerID="cri-o://e15e1f9275889bbcf74378e41ff25e0d837b167ba472c1d8b505af189d342494" gracePeriod=30
Oct 30 00:34:43.234057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.221657    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"]
Oct 30 00:34:43.234057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.221913    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj" podUID="c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" containerName="istio-proxy" containerID="cri-o://4526369ccd7d8c45da5281c742c3015ea2ff60105488ac69902eca9d29b5bc96" gracePeriod=30
Oct 30 00:34:43.352722 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.352686    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" event={"ID":"b66cdbd4-e6a1-44e2-b459-c1502f57225a","Type":"ContainerStarted","Data":"a24bdeccc59ccf13e33c9e421907d7cc82d5db34417ff3a315d0e43c5ee58611"}
Oct 30 00:34:43.501142 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.500449    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" event={"ID":"5c0fec19-a16c-40a1-8840-66022d000e75","Type":"ContainerStarted","Data":"cefadf30dfd6be3ea3fb5228ae0ff98396fb9437ddeadf9fbcd626e8f4ddcfe6"}
Oct 30 00:34:43.557700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.557668    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:34:43.561187 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.561156    4441 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 00:34:43.584193 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.566119    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 00:34:43.999050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.993466    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:34:43.999050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.993597    4441 patch_prober.go:29] interesting pod/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 00:34:43.999050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:43.993681    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 00:34:44.547048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:44.544386    4441 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 00:34:44.547048 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:44.544457    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 00:34:44.992684 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:44.992649    4441 patch_prober.go:29] interesting pod/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 00:34:44.992861 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:44.992712    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 00:34:45.537358 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:45.537319    4441 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 00:34:45.537636 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:45.537605    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 00:34:45.992617 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:45.992574    4441 patch_prober.go:29] interesting pod/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 00:34:45.993120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:45.992638    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 00:34:46.545443 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:46.545403    4441 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 00:34:46.545730 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:46.545697    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 00:34:46.974647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:46.974299    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:34:46.974647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:46.974374    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c" podUID="e9347cdc-b854-4c2f-b056-e01dd2e24b2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:34:46.991745 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:46.991717    4441 patch_prober.go:29] interesting pod/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 00:34:46.991888 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:46.991757    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 00:34:47.536037 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:47.535989    4441 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 00:34:47.536620 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:47.536590    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 00:34:47.570005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:47.569976    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:34:47.992434 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:47.992391    4441 patch_prober.go:29] interesting pod/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 00:34:47.992650 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:47.992454    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 00:34:48.303606 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.303486    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 00:34:48.303606 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.303547    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth" podUID="4a713f91-8d61-4d25-84d1-8121a511bd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 00:34:48.390686 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.390645    4441 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:34:48.390852 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.390712    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj" podUID="c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:34:48.583612 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.583572    4441 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 00:34:48.584150 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.583636    4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 00:34:48.623270 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.623207    4441 generic.go:356] "Generic (PLEG): container finished" podID="e9347cdc-b854-4c2f-b056-e01dd2e24b2c" containerID="e15e1f9275889bbcf74378e41ff25e0d837b167ba472c1d8b505af189d342494" exitCode=0
Oct 30 00:34:48.628239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.628128    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c" event={"ID":"e9347cdc-b854-4c2f-b056-e01dd2e24b2c","Type":"ContainerDied","Data":"e15e1f9275889bbcf74378e41ff25e0d837b167ba472c1d8b505af189d342494"}
Oct 30 00:34:48.631259 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.631159    4441 generic.go:356] "Generic (PLEG): container finished" podID="c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" containerID="4526369ccd7d8c45da5281c742c3015ea2ff60105488ac69902eca9d29b5bc96" exitCode=0
Oct 30 00:34:48.631691 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.631629    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj" event={"ID":"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd","Type":"ContainerDied","Data":"4526369ccd7d8c45da5281c742c3015ea2ff60105488ac69902eca9d29b5bc96"}
Oct 30 00:34:48.635516 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.635304    4441 generic.go:356] "Generic (PLEG): container finished" podID="4a713f91-8d61-4d25-84d1-8121a511bd94" containerID="1c31a27cb70a0863a5477e9973186b0540b45fcb2369d9996d6a405970730fdc" exitCode=0
Oct 30 00:34:48.635516 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.635478    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth" event={"ID":"4a713f91-8d61-4d25-84d1-8121a511bd94","Type":"ContainerDied","Data":"1c31a27cb70a0863a5477e9973186b0540b45fcb2369d9996d6a405970730fdc"}
Oct 30 00:34:48.792779 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.790569    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:34:48.797880 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.797794    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:34:48.839233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.839140    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-token\" (UniqueName: \"kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-token\") pod \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") "
Oct 30 00:34:48.839504 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.839482    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-certs\") pod \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") "
Oct 30 00:34:48.839666 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.839648    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-socket\") pod \"4a713f91-8d61-4d25-84d1-8121a511bd94\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") "
Oct 30 00:34:48.839852 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.839831    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-data\") pod \"4a713f91-8d61-4d25-84d1-8121a511bd94\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") "
Oct 30 00:34:48.840005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.839987    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istiod-ca-cert\") pod \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") "
Oct 30 00:34:48.840254 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.840233    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-kube-api-access-s7q27\" (UniqueName: \"kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-kube-api-access-s7q27\") pod \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") "
Oct 30 00:34:48.840417 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.840400    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a713f91-8d61-4d25-84d1-8121a511bd94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a713f91-8d61-4d25-84d1-8121a511bd94-istiod-ca-cert\") pod \"4a713f91-8d61-4d25-84d1-8121a511bd94\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") "
Oct 30 00:34:48.840552 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.840536    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-token\" (UniqueName: \"kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-token\") pod \"4a713f91-8d61-4d25-84d1-8121a511bd94\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") "
Oct 30 00:34:48.840734 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.840715    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-certs\") pod \"4a713f91-8d61-4d25-84d1-8121a511bd94\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") "
Oct 30 00:34:48.840860 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.840846    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-envoy\") pod \"4a713f91-8d61-4d25-84d1-8121a511bd94\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") "
Oct 30 00:34:48.841008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.840990    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-podinfo\") pod \"4a713f91-8d61-4d25-84d1-8121a511bd94\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") "
Oct 30 00:34:48.841180 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.841163    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-envoy\") pod \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") "
Oct 30 00:34:48.841347 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.841329    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-kube-api-access-h8zdl\" (UniqueName: \"kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-kube-api-access-h8zdl\") pod \"4a713f91-8d61-4d25-84d1-8121a511bd94\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") "
Oct 30 00:34:48.841511 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.841494    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-podinfo\") pod \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") "
Oct 30 00:34:48.841667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.841652    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-credential-socket\") pod \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") "
Oct 30 00:34:48.841853 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.841836    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-credential-socket\") pod \"4a713f91-8d61-4d25-84d1-8121a511bd94\" (UID: \"4a713f91-8d61-4d25-84d1-8121a511bd94\") "
Oct 30 00:34:48.842004 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.841988    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-data\") pod \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") "
Oct 30 00:34:48.842169 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.842152    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-socket\") pod \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\" (UID: \"e9347cdc-b854-4c2f-b056-e01dd2e24b2c\") "
Oct 30 00:34:48.843331 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.843308    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-socket" pod "e9347cdc-b854-4c2f-b056-e01dd2e24b2c" (UID: "e9347cdc-b854-4c2f-b056-e01dd2e24b2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:48.844641 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.844619    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-certs" pod "4a713f91-8d61-4d25-84d1-8121a511bd94" (UID: "4a713f91-8d61-4d25-84d1-8121a511bd94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:48.844971 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.844951    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-certs" pod "e9347cdc-b854-4c2f-b056-e01dd2e24b2c" (UID: "e9347cdc-b854-4c2f-b056-e01dd2e24b2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:48.845320 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.845300    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-socket" pod "4a713f91-8d61-4d25-84d1-8121a511bd94" (UID: "4a713f91-8d61-4d25-84d1-8121a511bd94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:48.845623 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.845605    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-data" pod "4a713f91-8d61-4d25-84d1-8121a511bd94" (UID: "4a713f91-8d61-4d25-84d1-8121a511bd94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:48.846056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.846018    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istiod-ca-cert" pod "e9347cdc-b854-4c2f-b056-e01dd2e24b2c" (UID: "e9347cdc-b854-4c2f-b056-e01dd2e24b2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:34:48.851429 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.851144    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-credential-socket" pod "e9347cdc-b854-4c2f-b056-e01dd2e24b2c" (UID: "e9347cdc-b854-4c2f-b056-e01dd2e24b2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:48.856871 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.856789    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-kube-api-access-s7q27" pod "e9347cdc-b854-4c2f-b056-e01dd2e24b2c" (UID: "e9347cdc-b854-4c2f-b056-e01dd2e24b2c"). InnerVolumeSpecName "kube-api-access-s7q27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:48.859391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.857906    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-token" pod "e9347cdc-b854-4c2f-b056-e01dd2e24b2c" (UID: "e9347cdc-b854-4c2f-b056-e01dd2e24b2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:48.859391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.857976    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-podinfo" pod "4a713f91-8d61-4d25-84d1-8121a511bd94" (UID: "4a713f91-8d61-4d25-84d1-8121a511bd94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:34:48.859391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.859197    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-data" pod "e9347cdc-b854-4c2f-b056-e01dd2e24b2c" (UID: "e9347cdc-b854-4c2f-b056-e01dd2e24b2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:48.860778 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.860745    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-podinfo" pod "e9347cdc-b854-4c2f-b056-e01dd2e24b2c" (UID: "e9347cdc-b854-4c2f-b056-e01dd2e24b2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:34:48.861420 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.861386    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a713f91-8d61-4d25-84d1-8121a511bd94-istiod-ca-cert" pod "4a713f91-8d61-4d25-84d1-8121a511bd94" (UID: "4a713f91-8d61-4d25-84d1-8121a511bd94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:34:48.861818 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.861799    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-credential-socket" pod "4a713f91-8d61-4d25-84d1-8121a511bd94" (UID: "4a713f91-8d61-4d25-84d1-8121a511bd94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:48.863096 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.863058    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-token" pod "4a713f91-8d61-4d25-84d1-8121a511bd94" (UID: "4a713f91-8d61-4d25-84d1-8121a511bd94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:48.865452 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.865420    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-envoy" pod "4a713f91-8d61-4d25-84d1-8121a511bd94" (UID: "4a713f91-8d61-4d25-84d1-8121a511bd94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:48.866535 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.866501    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-kube-api-access-h8zdl" pod "4a713f91-8d61-4d25-84d1-8121a511bd94" (UID: "4a713f91-8d61-4d25-84d1-8121a511bd94"). InnerVolumeSpecName "kube-api-access-h8zdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:48.877873 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.877834    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-envoy" pod "e9347cdc-b854-4c2f-b056-e01dd2e24b2c" (UID: "e9347cdc-b854-4c2f-b056-e01dd2e24b2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:48.946406 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946342    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946406 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946382    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946406 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946397    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946406 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946410    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946423    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946437    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8zdl\" (UniqueName: \"kubernetes.io/projected/4a713f91-8d61-4d25-84d1-8121a511bd94-kube-api-access-h8zdl\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946454    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946468    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946481    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946495    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946508    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946520    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946534    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946547    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946559    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a713f91-8d61-4d25-84d1-8121a511bd94-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946573    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946585    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7q27\" (UniqueName: \"kubernetes.io/projected/e9347cdc-b854-4c2f-b056-e01dd2e24b2c-kube-api-access-s7q27\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.946765 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.946598    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a713f91-8d61-4d25-84d1-8121a511bd94-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:48.976467 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.976438    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:34:48.996415 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.996388    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:34:48.996713 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.996693    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:34:48.997376 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:48.997357    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:34:49.046996 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.046964    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-kube-api-access-bpxlv\" (UniqueName: \"kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-kube-api-access-bpxlv\") pod \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") "
Oct 30 00:34:49.047188 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.047035    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-envoy\") pod \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") "
Oct 30 00:34:49.047188 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.047067    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-certs\") pod \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") "
Oct 30 00:34:49.047319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.047192    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-token\" (UniqueName: \"kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-token\") pod \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") "
Oct 30 00:34:49.047319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.047268    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istiod-ca-cert\") pod \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") "
Oct 30 00:34:49.047319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.047296    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-credential-socket\") pod \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") "
Oct 30 00:34:49.047466 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.047400    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-socket\") pod \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") "
Oct 30 00:34:49.047551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.047522    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-data\") pod \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") "
Oct 30 00:34:49.047626 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.047580    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-podinfo\") pod \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\" (UID: \"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd\") "
Oct 30 00:34:49.048782 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.048202    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-data" pod "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" (UID: "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:49.048782 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.048230    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-socket" pod "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" (UID: "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:49.048782 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.048142    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-certs" pod "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" (UID: "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:49.048782 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.048743    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-credential-socket" pod "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" (UID: "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:49.049104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.049062    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:49.049104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.049086    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:49.049104 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.049100    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:49.049233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.049116    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:49.049570 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.049544    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istiod-ca-cert" pod "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" (UID: "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:34:49.051778 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.051752    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-kube-api-access-bpxlv" pod "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" (UID: "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd"). InnerVolumeSpecName "kube-api-access-bpxlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:49.051910 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.051882    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-envoy" pod "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" (UID: "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:49.051982 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.051910    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-podinfo" pod "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" (UID: "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:34:49.053579 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.053553    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-token" pod "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" (UID: "c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:49.150867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.150789    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:49.150867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.150816    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:49.150867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.150826    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:49.150867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.150834    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:49.150867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.150842    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpxlv\" (UniqueName: \"kubernetes.io/projected/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd-kube-api-access-bpxlv\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:34:49.539937 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.539906    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:34:49.540951 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.540925    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:34:49.658701 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.658664    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj" event={"ID":"c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd","Type":"ContainerDied","Data":"75fecb4775098526adcc20ccb8ee41c4654e08e1ac31933044b96abb151f4750"}
Oct 30 00:34:49.659371 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.659346    4441 scope.go:122] "RemoveContainer" containerID="4526369ccd7d8c45da5281c742c3015ea2ff60105488ac69902eca9d29b5bc96"
Oct 30 00:34:49.659642 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.658681    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"
Oct 30 00:34:49.660860 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.660835    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth" event={"ID":"4a713f91-8d61-4d25-84d1-8121a511bd94","Type":"ContainerDied","Data":"f3eb4e1bed681b2221aa61e9f44ddc9f61cbc4ca6658cdbdd8080bb56456f284"}
Oct 30 00:34:49.661071 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.660853    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"
Oct 30 00:34:49.665073 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.665020    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"
Oct 30 00:34:49.665482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.665452    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c" event={"ID":"e9347cdc-b854-4c2f-b056-e01dd2e24b2c","Type":"ContainerDied","Data":"93c807cd02ffc06ca35ec6e0900e80a3faf93174aa24f9eb298d521a0fcc8235"}
Oct 30 00:34:49.692767 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.691403    4441 scope.go:122] "RemoveContainer" containerID="1c31a27cb70a0863a5477e9973186b0540b45fcb2369d9996d6a405970730fdc"
Oct 30 00:34:49.703919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.703897    4441 scope.go:122] "RemoveContainer" containerID="e15e1f9275889bbcf74378e41ff25e0d837b167ba472c1d8b505af189d342494"
Oct 30 00:34:49.749671 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.749636    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"]
Oct 30 00:34:49.755223 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.755191    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-6fwth"]
Oct 30 00:34:49.784179 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.784152    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"]
Oct 30 00:34:49.788622 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.788598    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-mfbnj"]
Oct 30 00:34:49.820798 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.820726    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"]
Oct 30 00:34:49.824308 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:49.824287    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-vnq2c"]
Oct 30 00:34:50.610423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:50.610391    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a713f91-8d61-4d25-84d1-8121a511bd94" path="/var/lib/kubelet/pods/4a713f91-8d61-4d25-84d1-8121a511bd94/volumes"
Oct 30 00:34:50.610864 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:50.610848    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" path="/var/lib/kubelet/pods/c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd/volumes"
Oct 30 00:34:50.611328 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:50.611311    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9347cdc-b854-4c2f-b056-e01dd2e24b2c" path="/var/lib/kubelet/pods/e9347cdc-b854-4c2f-b056-e01dd2e24b2c/volumes"
Oct 30 00:34:54.921359 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:54.921323    4441 scope.go:122] "RemoveContainer" containerID="6996d4f1ff17ccc6899a9e762d0804c606cc548c0fc937cbca2ef2620378ef1d"
Oct 30 00:34:54.933661 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:54.933634    4441 scope.go:122] "RemoveContainer" containerID="94ae8fff20855af9f41573af92274f8ac6f70c94dcdbf230b9dfe37a2288daf4"
Oct 30 00:34:54.945713 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:54.945688    4441 scope.go:122] "RemoveContainer" containerID="bd46f99662b1cd397cce54760b890fbab9b0fedb7fe86f1a7264e89df240f774"
Oct 30 00:34:56.916287 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:56.916245    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:34:56.922002 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:34:56.921980    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:35:02.214823 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:02.214789    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"]
Oct 30 00:35:02.247731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:02.247697    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"]
Oct 30 00:35:03.644973 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:03.644937    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:35:03.904900 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:35:03.904826    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:35:07.215336 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.215249    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq" podUID="1228f328-ccb3-4ed2-b77a-5f8018f50e77" containerName="payload-processing" containerID="cri-o://c3f3b684869e222d749bf83c40f4ce2c566360c7f4e2f3eb8d84ea9f7e30d0b9" gracePeriod=25
Oct 30 00:35:07.248427 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.248374    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5" podUID="1910466b-ea6b-4193-8242-e117d64a1f3e" containerName="payload-pre-processing" containerID="cri-o://57fb075e5020dc81992eb0b616414657ed784bf542b6bd85b6719fcea25a12e7" gracePeriod=25
Oct 30 00:35:07.574086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.574057    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"
Oct 30 00:35:07.615898 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.615862    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"
Oct 30 00:35:07.665184 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.665131    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1228f328-ccb3-4ed2-b77a-5f8018f50e77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1228f328-ccb3-4ed2-b77a-5f8018f50e77-plugins-config-volume\") pod \"1228f328-ccb3-4ed2-b77a-5f8018f50e77\" (UID: \"1228f328-ccb3-4ed2-b77a-5f8018f50e77\") "
Oct 30 00:35:07.665492 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.665459    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1910466b-ea6b-4193-8242-e117d64a1f3e-kube-api-access-kg48v\" (UniqueName: \"kubernetes.io/projected/1910466b-ea6b-4193-8242-e117d64a1f3e-kube-api-access-kg48v\") pod \"1910466b-ea6b-4193-8242-e117d64a1f3e\" (UID: \"1910466b-ea6b-4193-8242-e117d64a1f3e\") "
Oct 30 00:35:07.665724 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.665684    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1910466b-ea6b-4193-8242-e117d64a1f3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1910466b-ea6b-4193-8242-e117d64a1f3e-plugins-config-volume\") pod \"1910466b-ea6b-4193-8242-e117d64a1f3e\" (UID: \"1910466b-ea6b-4193-8242-e117d64a1f3e\") "
Oct 30 00:35:07.665957 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.665915    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1228f328-ccb3-4ed2-b77a-5f8018f50e77-kube-api-access-x8xr5\" (UniqueName: \"kubernetes.io/projected/1228f328-ccb3-4ed2-b77a-5f8018f50e77-kube-api-access-x8xr5\") pod \"1228f328-ccb3-4ed2-b77a-5f8018f50e77\" (UID: \"1228f328-ccb3-4ed2-b77a-5f8018f50e77\") "
Oct 30 00:35:07.667762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.667720    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1228f328-ccb3-4ed2-b77a-5f8018f50e77-plugins-config-volume" pod "1228f328-ccb3-4ed2-b77a-5f8018f50e77" (UID: "1228f328-ccb3-4ed2-b77a-5f8018f50e77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:07.668746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.668726    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1910466b-ea6b-4193-8242-e117d64a1f3e-plugins-config-volume" pod "1910466b-ea6b-4193-8242-e117d64a1f3e" (UID: "1910466b-ea6b-4193-8242-e117d64a1f3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:07.672419 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.672397    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1228f328-ccb3-4ed2-b77a-5f8018f50e77-kube-api-access-x8xr5" pod "1228f328-ccb3-4ed2-b77a-5f8018f50e77" (UID: "1228f328-ccb3-4ed2-b77a-5f8018f50e77"). InnerVolumeSpecName "kube-api-access-x8xr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:35:07.674479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.674455    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1910466b-ea6b-4193-8242-e117d64a1f3e-kube-api-access-kg48v" pod "1910466b-ea6b-4193-8242-e117d64a1f3e" (UID: "1910466b-ea6b-4193-8242-e117d64a1f3e"). InnerVolumeSpecName "kube-api-access-kg48v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:35:07.768385 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.768303    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1228f328-ccb3-4ed2-b77a-5f8018f50e77-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:35:07.768385 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.768331    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg48v\" (UniqueName: \"kubernetes.io/projected/1910466b-ea6b-4193-8242-e117d64a1f3e-kube-api-access-kg48v\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:35:07.768385 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.768367    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1910466b-ea6b-4193-8242-e117d64a1f3e-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:35:07.768385 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.768379    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8xr5\" (UniqueName: \"kubernetes.io/projected/1228f328-ccb3-4ed2-b77a-5f8018f50e77-kube-api-access-x8xr5\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:35:07.839415 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.839388    4441 generic.go:356] "Generic (PLEG): container finished" podID="1910466b-ea6b-4193-8242-e117d64a1f3e" containerID="57fb075e5020dc81992eb0b616414657ed784bf542b6bd85b6719fcea25a12e7" exitCode=0
Oct 30 00:35:07.839584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.839438    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"
Oct 30 00:35:07.839584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.839472    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5" event={"ID":"1910466b-ea6b-4193-8242-e117d64a1f3e","Type":"ContainerDied","Data":"57fb075e5020dc81992eb0b616414657ed784bf542b6bd85b6719fcea25a12e7"}
Oct 30 00:35:07.839584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.839523    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5" event={"ID":"1910466b-ea6b-4193-8242-e117d64a1f3e","Type":"ContainerDied","Data":"de1524e899714fc4feec10b88820ce96f61489cba790118d009e068053e1e2c0"}
Oct 30 00:35:07.839584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.839552    4441 scope.go:122] "RemoveContainer" containerID="57fb075e5020dc81992eb0b616414657ed784bf542b6bd85b6719fcea25a12e7"
Oct 30 00:35:07.841237 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.841213    4441 generic.go:356] "Generic (PLEG): container finished" podID="1228f328-ccb3-4ed2-b77a-5f8018f50e77" containerID="c3f3b684869e222d749bf83c40f4ce2c566360c7f4e2f3eb8d84ea9f7e30d0b9" exitCode=0
Oct 30 00:35:07.841367 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.841256    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"
Oct 30 00:35:07.841433 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.841394    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq" event={"ID":"1228f328-ccb3-4ed2-b77a-5f8018f50e77","Type":"ContainerDied","Data":"c3f3b684869e222d749bf83c40f4ce2c566360c7f4e2f3eb8d84ea9f7e30d0b9"}
Oct 30 00:35:07.841484 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.841454    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq" event={"ID":"1228f328-ccb3-4ed2-b77a-5f8018f50e77","Type":"ContainerDied","Data":"1b605cebd8fa7208e03ecaae72ee5a57f38ac7a7bcb39d21b1d374ef822afa77"}
Oct 30 00:35:07.853299 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.853220    4441 scope.go:122] "RemoveContainer" containerID="57fb075e5020dc81992eb0b616414657ed784bf542b6bd85b6719fcea25a12e7"
Oct 30 00:35:07.853724 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:35:07.853702    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57fb075e5020dc81992eb0b616414657ed784bf542b6bd85b6719fcea25a12e7\": container with ID starting with 57fb075e5020dc81992eb0b616414657ed784bf542b6bd85b6719fcea25a12e7 not found: ID does not exist" containerID="57fb075e5020dc81992eb0b616414657ed784bf542b6bd85b6719fcea25a12e7"
Oct 30 00:35:07.853825 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.853730    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57fb075e5020dc81992eb0b616414657ed784bf542b6bd85b6719fcea25a12e7"} err="failed to get container status \"57fb075e5020dc81992eb0b616414657ed784bf542b6bd85b6719fcea25a12e7\": rpc error: code = NotFound desc = could not find container \"57fb075e5020dc81992eb0b616414657ed784bf542b6bd85b6719fcea25a12e7\": container with ID starting with 57fb075e5020dc81992eb0b616414657ed784bf542b6bd85b6719fcea25a12e7 not found: ID does not exist"
Oct 30 00:35:07.853825 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.853747    4441 scope.go:122] "RemoveContainer" containerID="c3f3b684869e222d749bf83c40f4ce2c566360c7f4e2f3eb8d84ea9f7e30d0b9"
Oct 30 00:35:07.864565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.864545    4441 scope.go:122] "RemoveContainer" containerID="c3f3b684869e222d749bf83c40f4ce2c566360c7f4e2f3eb8d84ea9f7e30d0b9"
Oct 30 00:35:07.864844 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:35:07.864826    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f3b684869e222d749bf83c40f4ce2c566360c7f4e2f3eb8d84ea9f7e30d0b9\": container with ID starting with c3f3b684869e222d749bf83c40f4ce2c566360c7f4e2f3eb8d84ea9f7e30d0b9 not found: ID does not exist" containerID="c3f3b684869e222d749bf83c40f4ce2c566360c7f4e2f3eb8d84ea9f7e30d0b9"
Oct 30 00:35:07.864927 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.864850    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f3b684869e222d749bf83c40f4ce2c566360c7f4e2f3eb8d84ea9f7e30d0b9"} err="failed to get container status \"c3f3b684869e222d749bf83c40f4ce2c566360c7f4e2f3eb8d84ea9f7e30d0b9\": rpc error: code = NotFound desc = could not find container \"c3f3b684869e222d749bf83c40f4ce2c566360c7f4e2f3eb8d84ea9f7e30d0b9\": container with ID starting with c3f3b684869e222d749bf83c40f4ce2c566360c7f4e2f3eb8d84ea9f7e30d0b9 not found: ID does not exist"
Oct 30 00:35:07.891184 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.891162    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"]
Oct 30 00:35:07.897015 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.896986    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-be1f39-5fd59665d6-vklwq"]
Oct 30 00:35:07.935559 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.935099    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"]
Oct 30 00:35:07.938190 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:07.938163    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-be1f39-76596cc59-h7hc5"]
Oct 30 00:35:08.611589 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:08.611520    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1228f328-ccb3-4ed2-b77a-5f8018f50e77" path="/var/lib/kubelet/pods/1228f328-ccb3-4ed2-b77a-5f8018f50e77/volumes"
Oct 30 00:35:08.612168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:08.612008    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1910466b-ea6b-4193-8242-e117d64a1f3e" path="/var/lib/kubelet/pods/1910466b-ea6b-4193-8242-e117d64a1f3e/volumes"
Oct 30 00:35:22.828947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:22.828914    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"]
Oct 30 00:35:22.829519 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:22.829103    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd" podUID="6a472f11-984e-4097-9017-b9f8e7434853" containerName="maas-api" containerID="cri-o://7a671a14d31e8e2b72fbc77e0e9f3fadb892801802c6285e98884ed58c339c82" gracePeriod=30
Oct 30 00:35:22.952196 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:22.952161    4441 generic.go:356] "Generic (PLEG): container finished" podID="6a472f11-984e-4097-9017-b9f8e7434853" containerID="7a671a14d31e8e2b72fbc77e0e9f3fadb892801802c6285e98884ed58c339c82" exitCode=0
Oct 30 00:35:22.952323 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:22.952239    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd" event={"ID":"6a472f11-984e-4097-9017-b9f8e7434853","Type":"ContainerDied","Data":"7a671a14d31e8e2b72fbc77e0e9f3fadb892801802c6285e98884ed58c339c82"}
Oct 30 00:35:23.204714 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.204351    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:35:23.314603 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.314570    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-metrics-certs\") pod \"6a472f11-984e-4097-9017-b9f8e7434853\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") "
Oct 30 00:35:23.314795 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.314773    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-maas-api-tls\") pod \"6a472f11-984e-4097-9017-b9f8e7434853\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") "
Oct 30 00:35:23.314917 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.314885    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a472f11-984e-4097-9017-b9f8e7434853-kube-api-access-rfnzr\" (UniqueName: \"kubernetes.io/projected/6a472f11-984e-4097-9017-b9f8e7434853-kube-api-access-rfnzr\") pod \"6a472f11-984e-4097-9017-b9f8e7434853\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") "
Oct 30 00:35:23.315254 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.315232    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a472f11-984e-4097-9017-b9f8e7434853-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a472f11-984e-4097-9017-b9f8e7434853-openshift-service-ca\") pod \"6a472f11-984e-4097-9017-b9f8e7434853\" (UID: \"6a472f11-984e-4097-9017-b9f8e7434853\") "
Oct 30 00:35:23.315646 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.315620    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a472f11-984e-4097-9017-b9f8e7434853-openshift-service-ca" pod "6a472f11-984e-4097-9017-b9f8e7434853" (UID: "6a472f11-984e-4097-9017-b9f8e7434853"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:23.315853 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.315837    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a472f11-984e-4097-9017-b9f8e7434853-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:35:23.317108 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.317076    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-metrics-certs" pod "6a472f11-984e-4097-9017-b9f8e7434853" (UID: "6a472f11-984e-4097-9017-b9f8e7434853"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:23.317245 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.317228    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-maas-api-tls" pod "6a472f11-984e-4097-9017-b9f8e7434853" (UID: "6a472f11-984e-4097-9017-b9f8e7434853"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:23.319405 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.319379    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a472f11-984e-4097-9017-b9f8e7434853-kube-api-access-rfnzr" pod "6a472f11-984e-4097-9017-b9f8e7434853" (UID: "6a472f11-984e-4097-9017-b9f8e7434853"). InnerVolumeSpecName "kube-api-access-rfnzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:35:23.417603 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.417520    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:35:23.417603 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.417553    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfnzr\" (UniqueName: \"kubernetes.io/projected/6a472f11-984e-4097-9017-b9f8e7434853-kube-api-access-rfnzr\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:35:23.417603 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.417567    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a472f11-984e-4097-9017-b9f8e7434853-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:35:23.962012 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.961971    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd" event={"ID":"6a472f11-984e-4097-9017-b9f8e7434853","Type":"ContainerDied","Data":"5c35522efff41642b6ead3c4ce03b40f1eb65ba8f2046f1ad6b0e0abf9f44c6f"}
Oct 30 00:35:23.962515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.962012    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"
Oct 30 00:35:23.962515 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:23.962088    4441 scope.go:122] "RemoveContainer" containerID="7a671a14d31e8e2b72fbc77e0e9f3fadb892801802c6285e98884ed58c339c82"
Oct 30 00:35:24.014246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:24.014220    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"]
Oct 30 00:35:24.027448 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:24.027421    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-be1f39-65cddf6898-899gd"]
Oct 30 00:35:24.609514 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:24.609480    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a472f11-984e-4097-9017-b9f8e7434853" path="/var/lib/kubelet/pods/6a472f11-984e-4097-9017-b9f8e7434853/volumes"
Oct 30 00:35:26.026213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:26.026180    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"]
Oct 30 00:35:26.039547 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:26.039514    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f64499a1-jlr8r"]
Oct 30 00:35:26.613008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:26.612969    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2557020-c735-4e8e-9d2a-09bc294079e6" path="/var/lib/kubelet/pods/b2557020-c735-4e8e-9d2a-09bc294079e6/volumes"
Oct 30 00:35:33.902863 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:35:33.902826    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:35:45.380605 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:35:45.380567    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:36:03.903167 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:36:03.903120    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:36:08.420875 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.420841    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"]
Oct 30 00:36:08.421738 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421717    4441 cpu_manager.go:414] "containerMap: removing container" podUID="4a713f91-8d61-4d25-84d1-8121a511bd94" containerName="istio-proxy"
Oct 30 00:36:08.421787 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421743    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a713f91-8d61-4d25-84d1-8121a511bd94" containerName="istio-proxy"
Oct 30 00:36:08.421787 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421764    4441 cpu_manager.go:414] "containerMap: removing container" podUID="c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" containerName="istio-proxy"
Oct 30 00:36:08.421787 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421773    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" containerName="istio-proxy"
Oct 30 00:36:08.421892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421797    4441 cpu_manager.go:414] "containerMap: removing container" podUID="6a472f11-984e-4097-9017-b9f8e7434853" containerName="maas-api"
Oct 30 00:36:08.421892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421806    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a472f11-984e-4097-9017-b9f8e7434853" containerName="maas-api"
Oct 30 00:36:08.421892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421838    4441 cpu_manager.go:414] "containerMap: removing container" podUID="1228f328-ccb3-4ed2-b77a-5f8018f50e77" containerName="payload-processing"
Oct 30 00:36:08.421892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421845    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1228f328-ccb3-4ed2-b77a-5f8018f50e77" containerName="payload-processing"
Oct 30 00:36:08.421892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421858    4441 cpu_manager.go:414] "containerMap: removing container" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy"
Oct 30 00:36:08.421892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421865    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy"
Oct 30 00:36:08.421892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421878    4441 cpu_manager.go:414] "containerMap: removing container" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy"
Oct 30 00:36:08.421892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421885    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421902    4441 cpu_manager.go:414] "containerMap: removing container" podUID="1910466b-ea6b-4193-8242-e117d64a1f3e" containerName="payload-pre-processing"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421909    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1910466b-ea6b-4193-8242-e117d64a1f3e" containerName="payload-pre-processing"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421932    4441 cpu_manager.go:414] "containerMap: removing container" podUID="69a47c65-8652-4da3-a02d-961c57ff4d34" containerName="payload-processing"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421938    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a47c65-8652-4da3-a02d-961c57ff4d34" containerName="payload-processing"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421949    4441 cpu_manager.go:414] "containerMap: removing container" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421955    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421969    4441 cpu_manager.go:414] "containerMap: removing container" podUID="e9347cdc-b854-4c2f-b056-e01dd2e24b2c" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421976    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9347cdc-b854-4c2f-b056-e01dd2e24b2c" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421989    4441 cpu_manager.go:414] "containerMap: removing container" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.421996    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.422139    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a47c65-8652-4da3-a02d-961c57ff4d34" containerName="payload-processing"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.422152    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc27936a-64b1-473d-b857-07abf44fdacf" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.422164    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9347cdc-b854-4c2f-b056-e01dd2e24b2c" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.422174    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d355c2f5-010e-409c-81e7-8f1ff07db365" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.422184    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a713f91-8d61-4d25-84d1-8121a511bd94" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.422194    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2cdb408-2ee3-44a9-8926-1bb6cc8e0edd" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.422206    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a472f11-984e-4097-9017-b9f8e7434853" containerName="maas-api"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.422218    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1228f328-ccb3-4ed2-b77a-5f8018f50e77" containerName="payload-processing"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.422227    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.422235    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="daae17d4-1319-4ef7-adb7-a013358915db" containerName="istio-proxy"
Oct 30 00:36:08.422639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.422245    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1910466b-ea6b-4193-8242-e117d64a1f3e" containerName="payload-pre-processing"
Oct 30 00:36:08.451792 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.451762    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.457530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.457489    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"]
Oct 30 00:36:08.477460 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.477427    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-f1268c\"/\"openshift-service-ca.crt\""
Oct 30 00:36:08.477590 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.477501    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-f1268c\"/\"default-dockercfg-st4ww\""
Oct 30 00:36:08.477590 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.477503    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-f1268c\"/\"kube-root-ca.crt\""
Oct 30 00:36:08.477590 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.477511    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-f1268c\"/\"auth-test-model-f1268c-kserve-self-signed-certs\""
Oct 30 00:36:08.542070 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.542042    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rwzc\" (UniqueName: \"kubernetes.io/projected/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-kube-api-access-5rwzc\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.542232 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.542146    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tmp-dir\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.542311 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.542295    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-home\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.542356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.542330    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tls-certs\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.542510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.542486    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-dshm\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.542616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.542600    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-model-cache\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.643432 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.643390    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-home\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.643621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.643454    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tls-certs\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.643621 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.643537    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-dshm\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.643746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.643647    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-model-cache\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.643746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.643704    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rwzc\" (UniqueName: \"kubernetes.io/projected/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-kube-api-access-5rwzc\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.643855 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.643785    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tmp-dir\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.643914 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.643847    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-home\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.644186 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.644163    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-model-cache\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.644532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.644510    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tmp-dir\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.646513 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.646489    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-dshm\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.647065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.647010    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tls-certs\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.666480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.666453    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rwzc\" (UniqueName: \"kubernetes.io/projected/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-kube-api-access-5rwzc\") pod \"auth-test-model-f1268c-kserve-86f445cf7c-kfpch\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") " pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:08.774053 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:08.767769    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:09.121142 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:36:09.121107    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbaefb10_ab6e_4c79_b60d_f6c7e32bddf4.slice/crio-4b8d6c189712eb110aff7753f60daf60f68ede4beb75d0777923f758544f82a0 WatchSource:0}: Error finding container 4b8d6c189712eb110aff7753f60daf60f68ede4beb75d0777923f758544f82a0: Status 404 returned error can't find the container with id 4b8d6c189712eb110aff7753f60daf60f68ede4beb75d0777923f758544f82a0
Oct 30 00:36:09.121346 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:09.121241    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"]
Oct 30 00:36:09.362054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:09.361510    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch" event={"ID":"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4","Type":"ContainerStarted","Data":"90687fe38514e0b500003a7d2b4d6b7d9210d6ed5906a4a4344e6b64332a1a7a"}
Oct 30 00:36:09.362054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:09.361569    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch" event={"ID":"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4","Type":"ContainerStarted","Data":"4b8d6c189712eb110aff7753f60daf60f68ede4beb75d0777923f758544f82a0"}
Oct 30 00:36:09.403987 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:09.403879    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch" podStartSLOduration=1.403865675 podStartE2EDuration="1.403865675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:36:09.400830258 +0000 UTC m=+1878.926124136" watchObservedRunningTime="2026-10-30 00:36:09.403865675 +0000 UTC m=+1878.929159551"
Oct 30 00:36:09.768530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:09.768493    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:18.768272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:18.768239    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:18.782139 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:18.782105    4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:19.468966 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:19.468939    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:36:33.904502 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:36:33.904457    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:36:42.091936 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:42.091896    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:36:55.052114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:36:55.052080    4441 scope.go:122] "RemoveContainer" containerID="2150ebfce2f65f92c79723a3dda33ca8f2cc93c442043429f2a6aa7f27756834"
Oct 30 00:37:03.901191 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:37:03.901157    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:37:33.911319 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:37:33.911282    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:37:38.195306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:38.195267    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"]
Oct 30 00:37:38.258835 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:38.258796    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"]
Oct 30 00:37:38.259198 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:38.259167    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz" podUID="bc0d0e19-9703-41aa-8a3c-4e0a85057b94" containerName="maas-api" containerID="cri-o://27cf173a0cd31d6dcac7c491782c4411d4881c2629fa6e69e1cecd0c583a6442" gracePeriod=30
Oct 30 00:37:38.898874 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:38.898851    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:37:38.950978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:38.950943    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-openshift-service-ca\") pod \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") "
Oct 30 00:37:38.951406 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:38.951375    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-openshift-service-ca" pod "bc0d0e19-9703-41aa-8a3c-4e0a85057b94" (UID: "bc0d0e19-9703-41aa-8a3c-4e0a85057b94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:37:38.951550 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:38.951515    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-kube-api-access-pps9m\" (UniqueName: \"kubernetes.io/projected/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-kube-api-access-pps9m\") pod \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") "
Oct 30 00:37:38.951631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:38.951613    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-metrics-certs\") pod \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") "
Oct 30 00:37:38.951684 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:38.951671    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-maas-api-tls\") pod \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\" (UID: \"bc0d0e19-9703-41aa-8a3c-4e0a85057b94\") "
Oct 30 00:37:38.952280 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:38.952263    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:38.957286 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:38.957262    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-maas-api-tls" pod "bc0d0e19-9703-41aa-8a3c-4e0a85057b94" (UID: "bc0d0e19-9703-41aa-8a3c-4e0a85057b94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:37:38.957709 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:38.957682    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-metrics-certs" pod "bc0d0e19-9703-41aa-8a3c-4e0a85057b94" (UID: "bc0d0e19-9703-41aa-8a3c-4e0a85057b94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:37:38.958510 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:38.958477    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-kube-api-access-pps9m" pod "bc0d0e19-9703-41aa-8a3c-4e0a85057b94" (UID: "bc0d0e19-9703-41aa-8a3c-4e0a85057b94"). InnerVolumeSpecName "kube-api-access-pps9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:37:39.060057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:39.055749    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:39.060057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:39.055790    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:39.060057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:39.055806    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pps9m\" (UniqueName: \"kubernetes.io/projected/bc0d0e19-9703-41aa-8a3c-4e0a85057b94-kube-api-access-pps9m\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:39.074335 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:39.074295    4441 generic.go:356] "Generic (PLEG): container finished" podID="bc0d0e19-9703-41aa-8a3c-4e0a85057b94" containerID="27cf173a0cd31d6dcac7c491782c4411d4881c2629fa6e69e1cecd0c583a6442" exitCode=0
Oct 30 00:37:39.074518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:39.074354    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"
Oct 30 00:37:39.074518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:39.074366    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz" event={"ID":"bc0d0e19-9703-41aa-8a3c-4e0a85057b94","Type":"ContainerDied","Data":"27cf173a0cd31d6dcac7c491782c4411d4881c2629fa6e69e1cecd0c583a6442"}
Oct 30 00:37:39.074518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:39.074418    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz" event={"ID":"bc0d0e19-9703-41aa-8a3c-4e0a85057b94","Type":"ContainerDied","Data":"b000c9288d088aa12576e41d0ceec3ee07903cab44b1078d555b75e22dd230f0"}
Oct 30 00:37:39.074518 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:39.074443    4441 scope.go:122] "RemoveContainer" containerID="27cf173a0cd31d6dcac7c491782c4411d4881c2629fa6e69e1cecd0c583a6442"
Oct 30 00:37:39.093686 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:39.093654    4441 scope.go:122] "RemoveContainer" containerID="27cf173a0cd31d6dcac7c491782c4411d4881c2629fa6e69e1cecd0c583a6442"
Oct 30 00:37:39.094322 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:37:39.094270    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27cf173a0cd31d6dcac7c491782c4411d4881c2629fa6e69e1cecd0c583a6442\": container with ID starting with 27cf173a0cd31d6dcac7c491782c4411d4881c2629fa6e69e1cecd0c583a6442 not found: ID does not exist" containerID="27cf173a0cd31d6dcac7c491782c4411d4881c2629fa6e69e1cecd0c583a6442"
Oct 30 00:37:39.094454 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:39.094316    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27cf173a0cd31d6dcac7c491782c4411d4881c2629fa6e69e1cecd0c583a6442"} err="failed to get container status \"27cf173a0cd31d6dcac7c491782c4411d4881c2629fa6e69e1cecd0c583a6442\": rpc error: code = NotFound desc = could not find container \"27cf173a0cd31d6dcac7c491782c4411d4881c2629fa6e69e1cecd0c583a6442\": container with ID starting with 27cf173a0cd31d6dcac7c491782c4411d4881c2629fa6e69e1cecd0c583a6442 not found: ID does not exist"
Oct 30 00:37:39.140002 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:39.139911    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"]
Oct 30 00:37:39.145189 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:39.145155    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c1eeb6-5b897c78fc-b2mzz"]
Oct 30 00:37:40.243692 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:40.243655    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:37:40.613217 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:40.613143    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc0d0e19-9703-41aa-8a3c-4e0a85057b94" path="/var/lib/kubelet/pods/bc0d0e19-9703-41aa-8a3c-4e0a85057b94/volumes"
Oct 30 00:37:43.196529 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:43.196486    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml" podUID="02017bad-fe6f-415d-a24e-4b6b26a2a99a" containerName="payload-processing" containerID="cri-o://a076a8901284cf7c43af738b8ebb105160958bc58d64b8cba85a89247297c6e0" gracePeriod=25
Oct 30 00:37:43.746240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:43.746204    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"]
Oct 30 00:37:43.746614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:43.746499    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8" podUID="b4977042-bf8c-4bc6-8462-7f84acf9e5a2" containerName="maas-api" containerID="cri-o://128a15f0f851ca3483c759b4528263602b231c9d58f6149e0c764f4facf1f484" gracePeriod=30
Oct 30 00:37:44.196339 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.196296    4441 generic.go:356] "Generic (PLEG): container finished" podID="b4977042-bf8c-4bc6-8462-7f84acf9e5a2" containerID="128a15f0f851ca3483c759b4528263602b231c9d58f6149e0c764f4facf1f484" exitCode=0
Oct 30 00:37:44.196560 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.196507    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8" event={"ID":"b4977042-bf8c-4bc6-8462-7f84acf9e5a2","Type":"ContainerDied","Data":"128a15f0f851ca3483c759b4528263602b231c9d58f6149e0c764f4facf1f484"}
Oct 30 00:37:44.254072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.253044    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:37:44.351731 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.351674    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-metrics-certs\") pod \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") "
Oct 30 00:37:44.351959 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.351808    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-kube-api-access-rcxpj\" (UniqueName: \"kubernetes.io/projected/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-kube-api-access-rcxpj\") pod \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") "
Oct 30 00:37:44.356573 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.354570    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-openshift-service-ca\") pod \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") "
Oct 30 00:37:44.356573 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.355078    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-openshift-service-ca" pod "b4977042-bf8c-4bc6-8462-7f84acf9e5a2" (UID: "b4977042-bf8c-4bc6-8462-7f84acf9e5a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:37:44.356573 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.355703    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-maas-api-tls\") pod \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\" (UID: \"b4977042-bf8c-4bc6-8462-7f84acf9e5a2\") "
Oct 30 00:37:44.369109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.363228    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:44.369109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.368356    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-metrics-certs" pod "b4977042-bf8c-4bc6-8462-7f84acf9e5a2" (UID: "b4977042-bf8c-4bc6-8462-7f84acf9e5a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:37:44.369109 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.368524    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-kube-api-access-rcxpj" pod "b4977042-bf8c-4bc6-8462-7f84acf9e5a2" (UID: "b4977042-bf8c-4bc6-8462-7f84acf9e5a2"). InnerVolumeSpecName "kube-api-access-rcxpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:37:44.371521 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.371498    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-maas-api-tls" pod "b4977042-bf8c-4bc6-8462-7f84acf9e5a2" (UID: "b4977042-bf8c-4bc6-8462-7f84acf9e5a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:37:44.465680 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.465589    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:44.465680 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.465623    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcxpj\" (UniqueName: \"kubernetes.io/projected/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-kube-api-access-rcxpj\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:44.465680 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:44.465638    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4977042-bf8c-4bc6-8462-7f84acf9e5a2-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:45.205176 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:45.205145    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"
Oct 30 00:37:45.205176 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:45.205162    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8" event={"ID":"b4977042-bf8c-4bc6-8462-7f84acf9e5a2","Type":"ContainerDied","Data":"68adc4484d2cc2fde4cc9710b9a1bc5ee8d73835147f68af4486530eb2961620"}
Oct 30 00:37:45.205643 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:45.205257    4441 scope.go:122] "RemoveContainer" containerID="128a15f0f851ca3483c759b4528263602b231c9d58f6149e0c764f4facf1f484"
Oct 30 00:37:45.256313 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:45.256284    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"]
Oct 30 00:37:45.272227 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:45.272194    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a7ac6-5497f5c5bb-ppnn8"]
Oct 30 00:37:46.647059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:46.646647    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4977042-bf8c-4bc6-8462-7f84acf9e5a2" path="/var/lib/kubelet/pods/b4977042-bf8c-4bc6-8462-7f84acf9e5a2/volumes"
Oct 30 00:37:51.347968 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:51.347934    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml" podUID="02017bad-fe6f-415d-a24e-4b6b26a2a99a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:37:52.488356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:52.488320    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"]
Oct 30 00:37:52.505823 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:52.492320    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" podUID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" containerName="istio-proxy" containerID="cri-o://07eff54e2c6dad24f82f32fb4d687818072a77095b7d55cf47697ea209de4875" gracePeriod=30
Oct 30 00:37:52.505823 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:52.497500    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"]
Oct 30 00:37:52.505823 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:52.497672    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy" containerID="cri-o://61d91671bdb37f5a84d6ee053c1e342c3cde7609e11eb5dee09bccc33ac95caa" gracePeriod=30
Oct 30 00:37:52.514794 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:52.514764    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"]
Oct 30 00:37:52.515006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:52.514960    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd" podUID="2ae884f6-aa1a-4aca-addf-a0c37c26aa80" containerName="istio-proxy" containerID="cri-o://7642a7dcb345247fd59c1df969cdb375e69e5edda154845f359a47783367470a" gracePeriod=30
Oct 30 00:37:57.887140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.883436    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:37:57.925516 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.925487    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:37:57.951834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.951800    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-certs\") pod \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") "
Oct 30 00:37:57.951994 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.951872    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-kube-api-access-hps5z\" (UniqueName: \"kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-kube-api-access-hps5z\") pod \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") "
Oct 30 00:37:57.951994 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.951930    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-token\" (UniqueName: \"kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-token\") pod \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") "
Oct 30 00:37:57.952149 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.952006    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istiod-ca-cert\") pod \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") "
Oct 30 00:37:57.952225 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.952151    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-envoy\") pod \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") "
Oct 30 00:37:57.952284 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.952232    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-podinfo\") pod \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") "
Oct 30 00:37:57.952342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.952288    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-socket\") pod \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") "
Oct 30 00:37:57.952408 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.952357    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-data\") pod \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") "
Oct 30 00:37:57.952466 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.952410    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-credential-socket\") pod \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\" (UID: \"1f550b5f-f099-42d9-bd19-66ec69fce6d5\") "
Oct 30 00:37:57.952827 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.952793    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istiod-ca-cert" pod "1f550b5f-f099-42d9-bd19-66ec69fce6d5" (UID: "1f550b5f-f099-42d9-bd19-66ec69fce6d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:37:57.953266 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.953238    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-credential-socket" pod "1f550b5f-f099-42d9-bd19-66ec69fce6d5" (UID: "1f550b5f-f099-42d9-bd19-66ec69fce6d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:57.953561 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.953534    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:57.953651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.953561    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:57.955019 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.954920    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-data" pod "1f550b5f-f099-42d9-bd19-66ec69fce6d5" (UID: "1f550b5f-f099-42d9-bd19-66ec69fce6d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:57.955019 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.954992    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-socket" pod "1f550b5f-f099-42d9-bd19-66ec69fce6d5" (UID: "1f550b5f-f099-42d9-bd19-66ec69fce6d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:57.957941 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.957910    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-podinfo" pod "1f550b5f-f099-42d9-bd19-66ec69fce6d5" (UID: "1f550b5f-f099-42d9-bd19-66ec69fce6d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:37:57.958557 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.958529    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-certs" pod "1f550b5f-f099-42d9-bd19-66ec69fce6d5" (UID: "1f550b5f-f099-42d9-bd19-66ec69fce6d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:57.959335 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.959163    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-envoy" pod "1f550b5f-f099-42d9-bd19-66ec69fce6d5" (UID: "1f550b5f-f099-42d9-bd19-66ec69fce6d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:57.960553 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.960527    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-token" pod "1f550b5f-f099-42d9-bd19-66ec69fce6d5" (UID: "1f550b5f-f099-42d9-bd19-66ec69fce6d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:37:57.960644 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:57.960571    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-kube-api-access-hps5z" pod "1f550b5f-f099-42d9-bd19-66ec69fce6d5" (UID: "1f550b5f-f099-42d9-bd19-66ec69fce6d5"). InnerVolumeSpecName "kube-api-access-hps5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.070051    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-certs\") pod \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") "
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.070116    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-podinfo\") pod \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") "
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.070185    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-token\" (UniqueName: \"kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-token\") pod \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") "
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.070228    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-envoy\") pod \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") "
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.070281    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-credential-socket\") pod \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") "
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.070314    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-kube-api-access-rx2nq\" (UniqueName: \"kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-kube-api-access-rx2nq\") pod \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") "
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.070351    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-socket\") pod \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") "
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.070691    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istiod-ca-cert\") pod \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") "
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.070889    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-data\") pod \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\" (UID: \"2ae884f6-aa1a-4aca-addf-a0c37c26aa80\") "
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.071578    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.071598    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hps5z\" (UniqueName: \"kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-kube-api-access-hps5z\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.071609    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.071617    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.071624    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.071632    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.071640    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f550b5f-f099-42d9-bd19-66ec69fce6d5-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.071860    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-data" pod "2ae884f6-aa1a-4aca-addf-a0c37c26aa80" (UID: "2ae884f6-aa1a-4aca-addf-a0c37c26aa80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.072039    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-certs" pod "2ae884f6-aa1a-4aca-addf-a0c37c26aa80" (UID: "2ae884f6-aa1a-4aca-addf-a0c37c26aa80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.072555    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-credential-socket" pod "2ae884f6-aa1a-4aca-addf-a0c37c26aa80" (UID: "2ae884f6-aa1a-4aca-addf-a0c37c26aa80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.073431    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-socket" pod "2ae884f6-aa1a-4aca-addf-a0c37c26aa80" (UID: "2ae884f6-aa1a-4aca-addf-a0c37c26aa80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.073805    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istiod-ca-cert" pod "2ae884f6-aa1a-4aca-addf-a0c37c26aa80" (UID: "2ae884f6-aa1a-4aca-addf-a0c37c26aa80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:37:58.076750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.075653    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-podinfo" pod "2ae884f6-aa1a-4aca-addf-a0c37c26aa80" (UID: "2ae884f6-aa1a-4aca-addf-a0c37c26aa80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:37:58.088160 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.088130    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-token" pod "2ae884f6-aa1a-4aca-addf-a0c37c26aa80" (UID: "2ae884f6-aa1a-4aca-addf-a0c37c26aa80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:37:58.088642 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.088274    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-kube-api-access-rx2nq" pod "2ae884f6-aa1a-4aca-addf-a0c37c26aa80" (UID: "2ae884f6-aa1a-4aca-addf-a0c37c26aa80"). InnerVolumeSpecName "kube-api-access-rx2nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:37:58.092185 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.092122    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-envoy" pod "2ae884f6-aa1a-4aca-addf-a0c37c26aa80" (UID: "2ae884f6-aa1a-4aca-addf-a0c37c26aa80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:58.173182 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.173148    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.173182 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.173182    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.173361 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.173196    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.173361 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.173210    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.173361 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.173224    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.173361 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.173238    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.173361 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.173252    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx2nq\" (UniqueName: \"kubernetes.io/projected/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-kube-api-access-rx2nq\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.173361 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.173267    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.173361 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.173282    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae884f6-aa1a-4aca-addf-a0c37c26aa80-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.260900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.260873    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:37:58.312170 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.312136    4441 generic.go:356] "Generic (PLEG): container finished" podID="2ae884f6-aa1a-4aca-addf-a0c37c26aa80" containerID="7642a7dcb345247fd59c1df969cdb375e69e5edda154845f359a47783367470a" exitCode=0
Oct 30 00:37:58.312406 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.312207    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"
Oct 30 00:37:58.313061 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.312794    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd" event={"ID":"2ae884f6-aa1a-4aca-addf-a0c37c26aa80","Type":"ContainerDied","Data":"7642a7dcb345247fd59c1df969cdb375e69e5edda154845f359a47783367470a"}
Oct 30 00:37:58.313061 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.312886    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd" event={"ID":"2ae884f6-aa1a-4aca-addf-a0c37c26aa80","Type":"ContainerDied","Data":"bf8354449116dc8aec09f35624539f5975cd59d4af3d7729695f67999c9b2e4b"}
Oct 30 00:37:58.313061 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.312916    4441 scope.go:122] "RemoveContainer" containerID="7642a7dcb345247fd59c1df969cdb375e69e5edda154845f359a47783367470a"
Oct 30 00:37:58.315399 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.315375    4441 generic.go:356] "Generic (PLEG): container finished" podID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerID="61d91671bdb37f5a84d6ee053c1e342c3cde7609e11eb5dee09bccc33ac95caa" exitCode=0
Oct 30 00:37:58.315489 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.315471    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" event={"ID":"1f550b5f-f099-42d9-bd19-66ec69fce6d5","Type":"ContainerDied","Data":"61d91671bdb37f5a84d6ee053c1e342c3cde7609e11eb5dee09bccc33ac95caa"}
Oct 30 00:37:58.315647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.315514    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb" event={"ID":"1f550b5f-f099-42d9-bd19-66ec69fce6d5","Type":"ContainerDied","Data":"ec341f294515735be00bfd5973e677acfea37b9399dd9b0d3de5c41135308119"}
Oct 30 00:37:58.315647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.315609    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"
Oct 30 00:37:58.325391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.325317    4441 generic.go:356] "Generic (PLEG): container finished" podID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" containerID="07eff54e2c6dad24f82f32fb4d687818072a77095b7d55cf47697ea209de4875" exitCode=0
Oct 30 00:37:58.325617 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.325594    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"
Oct 30 00:37:58.325942 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.325914    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" event={"ID":"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5","Type":"ContainerDied","Data":"07eff54e2c6dad24f82f32fb4d687818072a77095b7d55cf47697ea209de4875"}
Oct 30 00:37:58.326119 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.325969    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj" event={"ID":"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5","Type":"ContainerDied","Data":"5f9d1400f31b775a8b94c59b0a3d742be6cdbc5f182424f7ceb5aea5d59849d5"}
Oct 30 00:37:58.332603 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.332538    4441 scope.go:122] "RemoveContainer" containerID="7642a7dcb345247fd59c1df969cdb375e69e5edda154845f359a47783367470a"
Oct 30 00:37:58.333312 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:37:58.333289    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7642a7dcb345247fd59c1df969cdb375e69e5edda154845f359a47783367470a\": container with ID starting with 7642a7dcb345247fd59c1df969cdb375e69e5edda154845f359a47783367470a not found: ID does not exist" containerID="7642a7dcb345247fd59c1df969cdb375e69e5edda154845f359a47783367470a"
Oct 30 00:37:58.333478 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.333439    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7642a7dcb345247fd59c1df969cdb375e69e5edda154845f359a47783367470a"} err="failed to get container status \"7642a7dcb345247fd59c1df969cdb375e69e5edda154845f359a47783367470a\": rpc error: code = NotFound desc = could not find container \"7642a7dcb345247fd59c1df969cdb375e69e5edda154845f359a47783367470a\": container with ID starting with 7642a7dcb345247fd59c1df969cdb375e69e5edda154845f359a47783367470a not found: ID does not exist"
Oct 30 00:37:58.333478 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.333473    4441 scope.go:122] "RemoveContainer" containerID="61d91671bdb37f5a84d6ee053c1e342c3cde7609e11eb5dee09bccc33ac95caa"
Oct 30 00:37:58.348019 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.347999    4441 scope.go:122] "RemoveContainer" containerID="5598768fe2770195b6b21bb83e54cde9c93eb45f006f47fe6486c49f975d280d"
Oct 30 00:37:58.359315 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.359275    4441 scope.go:122] "RemoveContainer" containerID="61d91671bdb37f5a84d6ee053c1e342c3cde7609e11eb5dee09bccc33ac95caa"
Oct 30 00:37:58.359646 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:37:58.359620    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d91671bdb37f5a84d6ee053c1e342c3cde7609e11eb5dee09bccc33ac95caa\": container with ID starting with 61d91671bdb37f5a84d6ee053c1e342c3cde7609e11eb5dee09bccc33ac95caa not found: ID does not exist" containerID="61d91671bdb37f5a84d6ee053c1e342c3cde7609e11eb5dee09bccc33ac95caa"
Oct 30 00:37:58.359708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.359656    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d91671bdb37f5a84d6ee053c1e342c3cde7609e11eb5dee09bccc33ac95caa"} err="failed to get container status \"61d91671bdb37f5a84d6ee053c1e342c3cde7609e11eb5dee09bccc33ac95caa\": rpc error: code = NotFound desc = could not find container \"61d91671bdb37f5a84d6ee053c1e342c3cde7609e11eb5dee09bccc33ac95caa\": container with ID starting with 61d91671bdb37f5a84d6ee053c1e342c3cde7609e11eb5dee09bccc33ac95caa not found: ID does not exist"
Oct 30 00:37:58.359708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.359678    4441 scope.go:122] "RemoveContainer" containerID="5598768fe2770195b6b21bb83e54cde9c93eb45f006f47fe6486c49f975d280d"
Oct 30 00:37:58.359987 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:37:58.359962    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5598768fe2770195b6b21bb83e54cde9c93eb45f006f47fe6486c49f975d280d\": container with ID starting with 5598768fe2770195b6b21bb83e54cde9c93eb45f006f47fe6486c49f975d280d not found: ID does not exist" containerID="5598768fe2770195b6b21bb83e54cde9c93eb45f006f47fe6486c49f975d280d"
Oct 30 00:37:58.360073 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.359996    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5598768fe2770195b6b21bb83e54cde9c93eb45f006f47fe6486c49f975d280d"} err="failed to get container status \"5598768fe2770195b6b21bb83e54cde9c93eb45f006f47fe6486c49f975d280d\": rpc error: code = NotFound desc = could not find container \"5598768fe2770195b6b21bb83e54cde9c93eb45f006f47fe6486c49f975d280d\": container with ID starting with 5598768fe2770195b6b21bb83e54cde9c93eb45f006f47fe6486c49f975d280d not found: ID does not exist"
Oct 30 00:37:58.360073 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.360019    4441 scope.go:122] "RemoveContainer" containerID="07eff54e2c6dad24f82f32fb4d687818072a77095b7d55cf47697ea209de4875"
Oct 30 00:37:58.369774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.369738    4441 scope.go:122] "RemoveContainer" containerID="07eff54e2c6dad24f82f32fb4d687818072a77095b7d55cf47697ea209de4875"
Oct 30 00:37:58.370118 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:37:58.370099    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07eff54e2c6dad24f82f32fb4d687818072a77095b7d55cf47697ea209de4875\": container with ID starting with 07eff54e2c6dad24f82f32fb4d687818072a77095b7d55cf47697ea209de4875 not found: ID does not exist" containerID="07eff54e2c6dad24f82f32fb4d687818072a77095b7d55cf47697ea209de4875"
Oct 30 00:37:58.370211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.370126    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07eff54e2c6dad24f82f32fb4d687818072a77095b7d55cf47697ea209de4875"} err="failed to get container status \"07eff54e2c6dad24f82f32fb4d687818072a77095b7d55cf47697ea209de4875\": rpc error: code = NotFound desc = could not find container \"07eff54e2c6dad24f82f32fb4d687818072a77095b7d55cf47697ea209de4875\": container with ID starting with 07eff54e2c6dad24f82f32fb4d687818072a77095b7d55cf47697ea209de4875 not found: ID does not exist"
Oct 30 00:37:58.375414 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.375389    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-certs\") pod \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") "
Oct 30 00:37:58.375533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.375445    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-credential-socket\") pod \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") "
Oct 30 00:37:58.375533 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.375480    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-podinfo\") pod \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") "
Oct 30 00:37:58.375658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.375535    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-data\") pod \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") "
Oct 30 00:37:58.375658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.375593    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-socket\") pod \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") "
Oct 30 00:37:58.375777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.375679    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-kube-api-access-9mrv6\" (UniqueName: \"kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-kube-api-access-9mrv6\") pod \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") "
Oct 30 00:37:58.375777 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.375740    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-token\" (UniqueName: \"kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-token\") pod \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") "
Oct 30 00:37:58.375886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.375798    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istiod-ca-cert\") pod \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") "
Oct 30 00:37:58.375886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.375835    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-envoy\") pod \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\" (UID: \"21c8a2b4-c0aa-41f2-a0af-80de573b6fb5\") "
Oct 30 00:37:58.376256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.376231    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-credential-socket" pod "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" (UID: "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:58.377130 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.376466    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-socket" pod "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" (UID: "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:58.377130 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.376513    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.377130 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.376667    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-data" pod "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" (UID: "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:58.377130 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.377085    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istiod-ca-cert" pod "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" (UID: "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:37:58.377808 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.377787    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-certs" pod "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" (UID: "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:58.378652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.378633    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-podinfo" pod "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" (UID: "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:37:58.378994 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.378967    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-envoy" pod "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" (UID: "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:58.380396 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.380373    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-kube-api-access-9mrv6" pod "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" (UID: "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5"). InnerVolumeSpecName "kube-api-access-9mrv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:37:58.380471 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.380451    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-token" pod "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" (UID: "21c8a2b4-c0aa-41f2-a0af-80de573b6fb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:37:58.394996 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.394972    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"]
Oct 30 00:37:58.400566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.400543    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-tlkwd"]
Oct 30 00:37:58.485050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.482128    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.485050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.482173    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.485050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.482187    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.485050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.482200    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.485050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.482214    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mrv6\" (UniqueName: \"kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-kube-api-access-9mrv6\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.485050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.482228    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.485050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.482241    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.485050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.482253    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:37:58.505183 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.501528    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"]
Oct 30 00:37:58.505183 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.501599    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-xzvtb"]
Oct 30 00:37:58.614045 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.613948    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" path="/var/lib/kubelet/pods/1f550b5f-f099-42d9-bd19-66ec69fce6d5/volumes"
Oct 30 00:37:58.614498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.614483    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae884f6-aa1a-4aca-addf-a0c37c26aa80" path="/var/lib/kubelet/pods/2ae884f6-aa1a-4aca-addf-a0c37c26aa80/volumes"
Oct 30 00:37:58.666556 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.666522    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"]
Oct 30 00:37:58.669760 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:37:58.669732    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-fsgfj"]
Oct 30 00:38:00.027325 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:00.027291    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"]
Oct 30 00:38:00.534304 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:00.534272    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"]
Oct 30 00:38:00.534820 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:00.534747    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2" podUID="4e076602-4664-4841-8894-75908287152d" containerName="maas-api" containerID="cri-o://a734ad563fc54178dc077791a19a834f10d12838d54969931da82201f771d0e8" gracePeriod=30
Oct 30 00:38:00.612206 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:00.612174    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" path="/var/lib/kubelet/pods/21c8a2b4-c0aa-41f2-a0af-80de573b6fb5/volumes"
Oct 30 00:38:01.063766 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.063580    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:38:01.141867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.141833    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e076602-4664-4841-8894-75908287152d-kube-api-access-8htpg\" (UniqueName: \"kubernetes.io/projected/4e076602-4664-4841-8894-75908287152d-kube-api-access-8htpg\") pod \"4e076602-4664-4841-8894-75908287152d\" (UID: \"4e076602-4664-4841-8894-75908287152d\") "
Oct 30 00:38:01.142093 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.141983    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-metrics-certs\") pod \"4e076602-4664-4841-8894-75908287152d\" (UID: \"4e076602-4664-4841-8894-75908287152d\") "
Oct 30 00:38:01.142196 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.142175    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e076602-4664-4841-8894-75908287152d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e076602-4664-4841-8894-75908287152d-openshift-service-ca\") pod \"4e076602-4664-4841-8894-75908287152d\" (UID: \"4e076602-4664-4841-8894-75908287152d\") "
Oct 30 00:38:01.142291 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.142274    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-maas-api-tls\") pod \"4e076602-4664-4841-8894-75908287152d\" (UID: \"4e076602-4664-4841-8894-75908287152d\") "
Oct 30 00:38:01.142657 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.142625    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e076602-4664-4841-8894-75908287152d-openshift-service-ca" pod "4e076602-4664-4841-8894-75908287152d" (UID: "4e076602-4664-4841-8894-75908287152d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:38:01.143291 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.143262    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e076602-4664-4841-8894-75908287152d-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:01.144759 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.144717    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-maas-api-tls" pod "4e076602-4664-4841-8894-75908287152d" (UID: "4e076602-4664-4841-8894-75908287152d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:38:01.144871 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.144773    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e076602-4664-4841-8894-75908287152d-kube-api-access-8htpg" pod "4e076602-4664-4841-8894-75908287152d" (UID: "4e076602-4664-4841-8894-75908287152d"). InnerVolumeSpecName "kube-api-access-8htpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:01.145270 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.145243    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-metrics-certs" pod "4e076602-4664-4841-8894-75908287152d" (UID: "4e076602-4664-4841-8894-75908287152d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:38:01.244383 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.244333    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:01.244383 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.244374    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8htpg\" (UniqueName: \"kubernetes.io/projected/4e076602-4664-4841-8894-75908287152d-kube-api-access-8htpg\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:01.244383 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.244388    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e076602-4664-4841-8894-75908287152d-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:01.347351 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.347262    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml" podUID="02017bad-fe6f-415d-a24e-4b6b26a2a99a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:38:01.353632 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.353600    4441 generic.go:356] "Generic (PLEG): container finished" podID="4e076602-4664-4841-8894-75908287152d" containerID="a734ad563fc54178dc077791a19a834f10d12838d54969931da82201f771d0e8" exitCode=0
Oct 30 00:38:01.353781 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.353642    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"
Oct 30 00:38:01.353781 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.353665    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2" event={"ID":"4e076602-4664-4841-8894-75908287152d","Type":"ContainerDied","Data":"a734ad563fc54178dc077791a19a834f10d12838d54969931da82201f771d0e8"}
Oct 30 00:38:01.353781 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.353709    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2" event={"ID":"4e076602-4664-4841-8894-75908287152d","Type":"ContainerDied","Data":"dedcff339cb4621266a438bed7cd7ac0439ab6dd0c1f13fdfe1f6f73cd208fed"}
Oct 30 00:38:01.353781 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.353734    4441 scope.go:122] "RemoveContainer" containerID="a734ad563fc54178dc077791a19a834f10d12838d54969931da82201f771d0e8"
Oct 30 00:38:01.365334 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.365308    4441 scope.go:122] "RemoveContainer" containerID="a734ad563fc54178dc077791a19a834f10d12838d54969931da82201f771d0e8"
Oct 30 00:38:01.365615 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:38:01.365597    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a734ad563fc54178dc077791a19a834f10d12838d54969931da82201f771d0e8\": container with ID starting with a734ad563fc54178dc077791a19a834f10d12838d54969931da82201f771d0e8 not found: ID does not exist" containerID="a734ad563fc54178dc077791a19a834f10d12838d54969931da82201f771d0e8"
Oct 30 00:38:01.365709 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.365622    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a734ad563fc54178dc077791a19a834f10d12838d54969931da82201f771d0e8"} err="failed to get container status \"a734ad563fc54178dc077791a19a834f10d12838d54969931da82201f771d0e8\": rpc error: code = NotFound desc = could not find container \"a734ad563fc54178dc077791a19a834f10d12838d54969931da82201f771d0e8\": container with ID starting with a734ad563fc54178dc077791a19a834f10d12838d54969931da82201f771d0e8 not found: ID does not exist"
Oct 30 00:38:01.403539 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.403508    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"]
Oct 30 00:38:01.407352 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:01.407329    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-29f3ca-5b759c9949-6stm2"]
Oct 30 00:38:02.609391 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:02.609362    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e076602-4664-4841-8894-75908287152d" path="/var/lib/kubelet/pods/4e076602-4664-4841-8894-75908287152d/volumes"
Oct 30 00:38:02.672161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:02.669501    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"]
Oct 30 00:38:02.672161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:02.669767    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl" podUID="58e7b78b-01af-4965-a2ee-0f73c0bf2e58" containerName="maas-api" containerID="cri-o://bffc3730df340cb3f57ed25e0cf63096a08b18ad3283e1b404a8c20fa8486618" gracePeriod=30
Oct 30 00:38:03.193330 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.193303    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:38:03.229724 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.229684    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-kube-api-access-j4tkn\" (UniqueName: \"kubernetes.io/projected/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-kube-api-access-j4tkn\") pod \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") "
Oct 30 00:38:03.229835 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.229820    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-openshift-service-ca\") pod \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") "
Oct 30 00:38:03.229881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.229862    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-maas-api-tls\") pod \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") "
Oct 30 00:38:03.229981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.229960    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-metrics-certs\") pod \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\" (UID: \"58e7b78b-01af-4965-a2ee-0f73c0bf2e58\") "
Oct 30 00:38:03.230341 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.230251    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-openshift-service-ca" pod "58e7b78b-01af-4965-a2ee-0f73c0bf2e58" (UID: "58e7b78b-01af-4965-a2ee-0f73c0bf2e58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:38:03.230655 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.230632    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:03.231814 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.231790    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-kube-api-access-j4tkn" pod "58e7b78b-01af-4965-a2ee-0f73c0bf2e58" (UID: "58e7b78b-01af-4965-a2ee-0f73c0bf2e58"). InnerVolumeSpecName "kube-api-access-j4tkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:03.232020 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.231997    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-maas-api-tls" pod "58e7b78b-01af-4965-a2ee-0f73c0bf2e58" (UID: "58e7b78b-01af-4965-a2ee-0f73c0bf2e58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:38:03.232331 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.232306    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-metrics-certs" pod "58e7b78b-01af-4965-a2ee-0f73c0bf2e58" (UID: "58e7b78b-01af-4965-a2ee-0f73c0bf2e58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:38:03.332111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.332078    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:03.332111 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.332113    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:03.332321 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.332130    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4tkn\" (UniqueName: \"kubernetes.io/projected/58e7b78b-01af-4965-a2ee-0f73c0bf2e58-kube-api-access-j4tkn\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:03.384757 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.384713    4441 generic.go:356] "Generic (PLEG): container finished" podID="02017bad-fe6f-415d-a24e-4b6b26a2a99a" containerID="a076a8901284cf7c43af738b8ebb105160958bc58d64b8cba85a89247297c6e0" exitCode=0
Oct 30 00:38:03.384912 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.384891    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml" event={"ID":"02017bad-fe6f-415d-a24e-4b6b26a2a99a","Type":"ContainerDied","Data":"a076a8901284cf7c43af738b8ebb105160958bc58d64b8cba85a89247297c6e0"}
Oct 30 00:38:03.386733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.386704    4441 generic.go:356] "Generic (PLEG): container finished" podID="58e7b78b-01af-4965-a2ee-0f73c0bf2e58" containerID="bffc3730df340cb3f57ed25e0cf63096a08b18ad3283e1b404a8c20fa8486618" exitCode=0
Oct 30 00:38:03.386892 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.386866    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl" event={"ID":"58e7b78b-01af-4965-a2ee-0f73c0bf2e58","Type":"ContainerDied","Data":"bffc3730df340cb3f57ed25e0cf63096a08b18ad3283e1b404a8c20fa8486618"}
Oct 30 00:38:03.387038 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.386919    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl" event={"ID":"58e7b78b-01af-4965-a2ee-0f73c0bf2e58","Type":"ContainerDied","Data":"cbbe777a38395deb8b09c9e4c7fb075ee453c361a5ad44285038381e6ec26be4"}
Oct 30 00:38:03.387038 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.386981    4441 scope.go:122] "RemoveContainer" containerID="bffc3730df340cb3f57ed25e0cf63096a08b18ad3283e1b404a8c20fa8486618"
Oct 30 00:38:03.387244 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.387191    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"
Oct 30 00:38:03.414714 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.414695    4441 scope.go:122] "RemoveContainer" containerID="bffc3730df340cb3f57ed25e0cf63096a08b18ad3283e1b404a8c20fa8486618"
Oct 30 00:38:03.415017 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:38:03.414990    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bffc3730df340cb3f57ed25e0cf63096a08b18ad3283e1b404a8c20fa8486618\": container with ID starting with bffc3730df340cb3f57ed25e0cf63096a08b18ad3283e1b404a8c20fa8486618 not found: ID does not exist" containerID="bffc3730df340cb3f57ed25e0cf63096a08b18ad3283e1b404a8c20fa8486618"
Oct 30 00:38:03.415116 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.415061    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bffc3730df340cb3f57ed25e0cf63096a08b18ad3283e1b404a8c20fa8486618"} err="failed to get container status \"bffc3730df340cb3f57ed25e0cf63096a08b18ad3283e1b404a8c20fa8486618\": rpc error: code = NotFound desc = could not find container \"bffc3730df340cb3f57ed25e0cf63096a08b18ad3283e1b404a8c20fa8486618\": container with ID starting with bffc3730df340cb3f57ed25e0cf63096a08b18ad3283e1b404a8c20fa8486618 not found: ID does not exist"
Oct 30 00:38:03.451461 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.451435    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"]
Oct 30 00:38:03.451664 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.451648    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ecb34a-75b487f886-6vtwl"]
Oct 30 00:38:03.519672 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.519645    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"
Oct 30 00:38:03.634534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.634499    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02017bad-fe6f-415d-a24e-4b6b26a2a99a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02017bad-fe6f-415d-a24e-4b6b26a2a99a-plugins-config-volume\") pod \"02017bad-fe6f-415d-a24e-4b6b26a2a99a\" (UID: \"02017bad-fe6f-415d-a24e-4b6b26a2a99a\") "
Oct 30 00:38:03.635080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.634564    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02017bad-fe6f-415d-a24e-4b6b26a2a99a-kube-api-access-gv7kg\" (UniqueName: \"kubernetes.io/projected/02017bad-fe6f-415d-a24e-4b6b26a2a99a-kube-api-access-gv7kg\") pod \"02017bad-fe6f-415d-a24e-4b6b26a2a99a\" (UID: \"02017bad-fe6f-415d-a24e-4b6b26a2a99a\") "
Oct 30 00:38:03.635080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.634913    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02017bad-fe6f-415d-a24e-4b6b26a2a99a-plugins-config-volume" pod "02017bad-fe6f-415d-a24e-4b6b26a2a99a" (UID: "02017bad-fe6f-415d-a24e-4b6b26a2a99a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:38:03.635267 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.635246    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02017bad-fe6f-415d-a24e-4b6b26a2a99a-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:03.636837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.636810    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02017bad-fe6f-415d-a24e-4b6b26a2a99a-kube-api-access-gv7kg" pod "02017bad-fe6f-415d-a24e-4b6b26a2a99a" (UID: "02017bad-fe6f-415d-a24e-4b6b26a2a99a"). InnerVolumeSpecName "kube-api-access-gv7kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:03.736919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:03.736890    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv7kg\" (UniqueName: \"kubernetes.io/projected/02017bad-fe6f-415d-a24e-4b6b26a2a99a-kube-api-access-gv7kg\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:03.901367 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:38:03.901325    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:38:04.393922 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:04.393889    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"
Oct 30 00:38:04.394144 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:04.393927    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml" event={"ID":"02017bad-fe6f-415d-a24e-4b6b26a2a99a","Type":"ContainerDied","Data":"f62bfc7303192f6401b741e8392d9b562715c213c823ef048894800c0caa5611"}
Oct 30 00:38:04.394144 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:04.393991    4441 scope.go:122] "RemoveContainer" containerID="a076a8901284cf7c43af738b8ebb105160958bc58d64b8cba85a89247297c6e0"
Oct 30 00:38:04.443164 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:04.443128    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"]
Oct 30 00:38:04.454416 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:04.454390    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c1eeb6-cb477b49b-ntgml"]
Oct 30 00:38:04.609569 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:04.609538    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02017bad-fe6f-415d-a24e-4b6b26a2a99a" path="/var/lib/kubelet/pods/02017bad-fe6f-415d-a24e-4b6b26a2a99a/volumes"
Oct 30 00:38:04.609980 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:04.609966    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e7b78b-01af-4965-a2ee-0f73c0bf2e58" path="/var/lib/kubelet/pods/58e7b78b-01af-4965-a2ee-0f73c0bf2e58/volumes"
Oct 30 00:38:05.028358 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:05.028314    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x" podUID="0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c" containerName="payload-pre-processing" containerID="cri-o://70fe7ce9f2e3e02325646ec405c8d7346bf768a63c4ecbf26cf5a11ac7a5a8ef" gracePeriod=25
Oct 30 00:38:06.295439 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:06.295404    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x" podUID="0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:38:10.246697 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:10.246658    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"]
Oct 30 00:38:10.247240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:10.246920    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerName="istio-proxy" containerID="cri-o://a24bdeccc59ccf13e33c9e421907d7cc82d5db34417ff3a315d0e43c5ee58611" gracePeriod=30
Oct 30 00:38:15.490600 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.490541    4441 generic.go:356] "Generic (PLEG): container finished" podID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerID="a24bdeccc59ccf13e33c9e421907d7cc82d5db34417ff3a315d0e43c5ee58611" exitCode=0
Oct 30 00:38:15.491357 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.490636    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" event={"ID":"b66cdbd4-e6a1-44e2-b459-c1502f57225a","Type":"ContainerDied","Data":"a24bdeccc59ccf13e33c9e421907d7cc82d5db34417ff3a315d0e43c5ee58611"}
Oct 30 00:38:15.491357 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.490694    4441 scope.go:122] "RemoveContainer" containerID="ed24d7c957fc8094b9eec347e8b9e6bedb792ad631904b3647b80e6034f41684"
Oct 30 00:38:15.593705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.593680    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:38:15.643659 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.643627    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istiod-ca-cert\") pod \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") "
Oct 30 00:38:15.643812 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.643730    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-certs\") pod \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") "
Oct 30 00:38:15.643893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.643811    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-kube-api-access-kpfds\" (UniqueName: \"kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-kube-api-access-kpfds\") pod \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") "
Oct 30 00:38:15.643893 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.643857    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-token\" (UniqueName: \"kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-token\") pod \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") "
Oct 30 00:38:15.644008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.643923    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-data\") pod \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") "
Oct 30 00:38:15.644008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.643986    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-envoy\") pod \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") "
Oct 30 00:38:15.644167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.644013    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-credential-socket\") pod \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") "
Oct 30 00:38:15.644167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.644049    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-podinfo\") pod \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") "
Oct 30 00:38:15.644167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.644078    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-socket\") pod \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\" (UID: \"b66cdbd4-e6a1-44e2-b459-c1502f57225a\") "
Oct 30 00:38:15.644167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.644087    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istiod-ca-cert" pod "b66cdbd4-e6a1-44e2-b459-c1502f57225a" (UID: "b66cdbd4-e6a1-44e2-b459-c1502f57225a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:38:15.644401 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.644369    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-certs" pod "b66cdbd4-e6a1-44e2-b459-c1502f57225a" (UID: "b66cdbd4-e6a1-44e2-b459-c1502f57225a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:15.644658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.644604    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-socket" pod "b66cdbd4-e6a1-44e2-b459-c1502f57225a" (UID: "b66cdbd4-e6a1-44e2-b459-c1502f57225a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:15.644658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.644633    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:15.644658 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.644656    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:15.644968 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.644942    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-data" pod "b66cdbd4-e6a1-44e2-b459-c1502f57225a" (UID: "b66cdbd4-e6a1-44e2-b459-c1502f57225a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:15.645088 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.645063    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-credential-socket" pod "b66cdbd4-e6a1-44e2-b459-c1502f57225a" (UID: "b66cdbd4-e6a1-44e2-b459-c1502f57225a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:15.646607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.646581    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-envoy" pod "b66cdbd4-e6a1-44e2-b459-c1502f57225a" (UID: "b66cdbd4-e6a1-44e2-b459-c1502f57225a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:15.646824 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.646800    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-token" pod "b66cdbd4-e6a1-44e2-b459-c1502f57225a" (UID: "b66cdbd4-e6a1-44e2-b459-c1502f57225a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:15.647341 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.646857    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-kube-api-access-kpfds" pod "b66cdbd4-e6a1-44e2-b459-c1502f57225a" (UID: "b66cdbd4-e6a1-44e2-b459-c1502f57225a"). InnerVolumeSpecName "kube-api-access-kpfds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:15.647998 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.647967    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-podinfo" pod "b66cdbd4-e6a1-44e2-b459-c1502f57225a" (UID: "b66cdbd4-e6a1-44e2-b459-c1502f57225a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:38:15.745665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.745624    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpfds\" (UniqueName: \"kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-kube-api-access-kpfds\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:15.745665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.745663    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:15.745881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.745679    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:15.745881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.745694    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:15.745881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.745709    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:15.745881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.745722    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66cdbd4-e6a1-44e2-b459-c1502f57225a-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:15.745881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.745736    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66cdbd4-e6a1-44e2-b459-c1502f57225a-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:15.746811 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:15.746782    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"]
Oct 30 00:38:16.295977 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:16.295935    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x" podUID="0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:38:16.497613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:16.497583    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"
Oct 30 00:38:16.498044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:16.497616    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9" event={"ID":"b66cdbd4-e6a1-44e2-b459-c1502f57225a","Type":"ContainerDied","Data":"acdefda43f741831f2844101b176a765f001ccaed1bb845c2f99e73c770e11bd"}
Oct 30 00:38:16.498044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:16.497698    4441 scope.go:122] "RemoveContainer" containerID="a24bdeccc59ccf13e33c9e421907d7cc82d5db34417ff3a315d0e43c5ee58611"
Oct 30 00:38:16.549500 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:16.549413    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"]
Oct 30 00:38:16.559692 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:16.559659    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ecb34a-openshift-default-69d6d6cf97-8fhf9"]
Oct 30 00:38:16.609815 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:16.609785    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" path="/var/lib/kubelet/pods/b66cdbd4-e6a1-44e2-b459-c1502f57225a/volumes"
Oct 30 00:38:21.506454 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:21.506418    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"]
Oct 30 00:38:22.147761 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.147709    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"]
Oct 30 00:38:22.148642 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.148605    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg" podUID="2f7ee5a9-ec28-41f7-8dcf-0a88595c3400" containerName="maas-api" containerID="cri-o://b6fabaf38b859ac8a0c951f6154619997531c23c7808fe966a925ffecd42beb8" gracePeriod=30
Oct 30 00:38:22.459100 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.446459    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg" podUID="2f7ee5a9-ec28-41f7-8dcf-0a88595c3400" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.158:8443/health\": dial tcp 10.133.0.158:8443: connect: connection refused"
Oct 30 00:38:22.600444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.600218    4441 generic.go:356] "Generic (PLEG): container finished" podID="2f7ee5a9-ec28-41f7-8dcf-0a88595c3400" containerID="b6fabaf38b859ac8a0c951f6154619997531c23c7808fe966a925ffecd42beb8" exitCode=0
Oct 30 00:38:22.600444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.600375    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg" event={"ID":"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400","Type":"ContainerDied","Data":"b6fabaf38b859ac8a0c951f6154619997531c23c7808fe966a925ffecd42beb8"}
Oct 30 00:38:22.751312 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.751290    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:38:22.826183 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.826150    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-openshift-service-ca\") pod \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") "
Oct 30 00:38:22.826361 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.826275    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-metrics-certs\") pod \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") "
Oct 30 00:38:22.826451 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.826400    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-kube-api-access-jkm7j\" (UniqueName: \"kubernetes.io/projected/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-kube-api-access-jkm7j\") pod \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") "
Oct 30 00:38:22.826540 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.826515    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-openshift-service-ca" pod "2f7ee5a9-ec28-41f7-8dcf-0a88595c3400" (UID: "2f7ee5a9-ec28-41f7-8dcf-0a88595c3400"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:38:22.829323 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.826953    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-maas-api-tls\") pod \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\" (UID: \"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400\") "
Oct 30 00:38:22.829323 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.827553    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:22.839242 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.839215    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-maas-api-tls" pod "2f7ee5a9-ec28-41f7-8dcf-0a88595c3400" (UID: "2f7ee5a9-ec28-41f7-8dcf-0a88595c3400"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:38:22.839242 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.839215    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-kube-api-access-jkm7j" pod "2f7ee5a9-ec28-41f7-8dcf-0a88595c3400" (UID: "2f7ee5a9-ec28-41f7-8dcf-0a88595c3400"). InnerVolumeSpecName "kube-api-access-jkm7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:22.845008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.843353    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-metrics-certs" pod "2f7ee5a9-ec28-41f7-8dcf-0a88595c3400" (UID: "2f7ee5a9-ec28-41f7-8dcf-0a88595c3400"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:38:22.929080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.929055    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:22.929080 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.929078    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkm7j\" (UniqueName: \"kubernetes.io/projected/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-kube-api-access-jkm7j\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:22.929239 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:22.929088    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:23.608602 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:23.608578    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"
Oct 30 00:38:23.609052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:23.608606    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg" event={"ID":"2f7ee5a9-ec28-41f7-8dcf-0a88595c3400","Type":"ContainerDied","Data":"85e32eacdfbd84480d4af81f0770ded7151ea3cd53782db6bd14dbedffda7a17"}
Oct 30 00:38:23.609052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:23.608663    4441 scope.go:122] "RemoveContainer" containerID="b6fabaf38b859ac8a0c951f6154619997531c23c7808fe966a925ffecd42beb8"
Oct 30 00:38:23.659083 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:23.659052    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"]
Oct 30 00:38:23.669740 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:23.669712    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a782e5-76d6dd77fc-bmfdg"]
Oct 30 00:38:24.609756 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:24.609723    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7ee5a9-ec28-41f7-8dcf-0a88595c3400" path="/var/lib/kubelet/pods/2f7ee5a9-ec28-41f7-8dcf-0a88595c3400/volumes"
Oct 30 00:38:25.331619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.331294    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"
Oct 30 00:38:25.365835 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.365806    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-kube-api-access-s4tzd\" (UniqueName: \"kubernetes.io/projected/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-kube-api-access-s4tzd\") pod \"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c\" (UID: \"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c\") "
Oct 30 00:38:25.366050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.366018    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-plugins-config-volume\") pod \"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c\" (UID: \"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c\") "
Oct 30 00:38:25.366405 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.366377    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-plugins-config-volume" pod "0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c" (UID: "0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:38:25.366551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.366536    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:25.368053 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.368015    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-kube-api-access-s4tzd" pod "0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c" (UID: "0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c"). InnerVolumeSpecName "kube-api-access-s4tzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:25.467945 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.467908    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4tzd\" (UniqueName: \"kubernetes.io/projected/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c-kube-api-access-s4tzd\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:25.624050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.623956    4441 generic.go:356] "Generic (PLEG): container finished" podID="0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c" containerID="70fe7ce9f2e3e02325646ec405c8d7346bf768a63c4ecbf26cf5a11ac7a5a8ef" exitCode=0
Oct 30 00:38:25.624050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.624017    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"
Oct 30 00:38:25.624540 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.624048    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x" event={"ID":"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c","Type":"ContainerDied","Data":"70fe7ce9f2e3e02325646ec405c8d7346bf768a63c4ecbf26cf5a11ac7a5a8ef"}
Oct 30 00:38:25.624540 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.624112    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x" event={"ID":"0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c","Type":"ContainerDied","Data":"c2caf619da3b7e4720560d185faa9efce1dd96032e9fdd4e166dc91209049f5b"}
Oct 30 00:38:25.624540 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.624135    4441 scope.go:122] "RemoveContainer" containerID="70fe7ce9f2e3e02325646ec405c8d7346bf768a63c4ecbf26cf5a11ac7a5a8ef"
Oct 30 00:38:25.634526 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.634510    4441 scope.go:122] "RemoveContainer" containerID="70fe7ce9f2e3e02325646ec405c8d7346bf768a63c4ecbf26cf5a11ac7a5a8ef"
Oct 30 00:38:25.634800 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:38:25.634781    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70fe7ce9f2e3e02325646ec405c8d7346bf768a63c4ecbf26cf5a11ac7a5a8ef\": container with ID starting with 70fe7ce9f2e3e02325646ec405c8d7346bf768a63c4ecbf26cf5a11ac7a5a8ef not found: ID does not exist" containerID="70fe7ce9f2e3e02325646ec405c8d7346bf768a63c4ecbf26cf5a11ac7a5a8ef"
Oct 30 00:38:25.634865 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.634811    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70fe7ce9f2e3e02325646ec405c8d7346bf768a63c4ecbf26cf5a11ac7a5a8ef"} err="failed to get container status \"70fe7ce9f2e3e02325646ec405c8d7346bf768a63c4ecbf26cf5a11ac7a5a8ef\": rpc error: code = NotFound desc = could not find container \"70fe7ce9f2e3e02325646ec405c8d7346bf768a63c4ecbf26cf5a11ac7a5a8ef\": container with ID starting with 70fe7ce9f2e3e02325646ec405c8d7346bf768a63c4ecbf26cf5a11ac7a5a8ef not found: ID does not exist"
Oct 30 00:38:25.674503 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.674475    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"]
Oct 30 00:38:25.685423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:25.685400    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-29f3ca-b76c9bf5f-m7g2x"]
Oct 30 00:38:26.507049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:26.506934    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm" podUID="759af21c-c080-4627-8863-b3f419cd4420" containerName="payload-pre-processing" containerID="cri-o://8a85a64d901ba20f90974954493e7cb45e4068bbc1fdbb6786a7874562504a2d" gracePeriod=25
Oct 30 00:38:26.610287 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:26.610256    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c" path="/var/lib/kubelet/pods/0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c/volumes"
Oct 30 00:38:30.772571 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:30.772534    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main" containerID="cri-o://bd04ecaccc0e151cd7d0602eb86388fa082c9ee85a108c2ef6ac99d4b6a9cdda" gracePeriod=45
Oct 30 00:38:31.060584 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.060557    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:38:31.150446 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.150406    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92906908-16f4-45e4-8177-1b53992469a5-kube-api-access-8txzf\" (UniqueName: \"kubernetes.io/projected/92906908-16f4-45e4-8177-1b53992469a5-kube-api-access-8txzf\") pod \"92906908-16f4-45e4-8177-1b53992469a5\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") "
Oct 30 00:38:31.150647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.150482    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-model-cache\") pod \"92906908-16f4-45e4-8177-1b53992469a5\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") "
Oct 30 00:38:31.150647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.150575    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-home\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-home\") pod \"92906908-16f4-45e4-8177-1b53992469a5\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") "
Oct 30 00:38:31.150647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.150630    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92906908-16f4-45e4-8177-1b53992469a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/92906908-16f4-45e4-8177-1b53992469a5-tls-certs\") pod \"92906908-16f4-45e4-8177-1b53992469a5\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") "
Oct 30 00:38:31.150841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.150664    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-dshm\") pod \"92906908-16f4-45e4-8177-1b53992469a5\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") "
Oct 30 00:38:31.150841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.150785    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-tmp-dir\") pod \"92906908-16f4-45e4-8177-1b53992469a5\" (UID: \"92906908-16f4-45e4-8177-1b53992469a5\") "
Oct 30 00:38:31.152067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.152015    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-model-cache" pod "92906908-16f4-45e4-8177-1b53992469a5" (UID: "92906908-16f4-45e4-8177-1b53992469a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:31.152235 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.152208    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-home" pod "92906908-16f4-45e4-8177-1b53992469a5" (UID: "92906908-16f4-45e4-8177-1b53992469a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:31.152462 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.152440    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-tmp-dir" pod "92906908-16f4-45e4-8177-1b53992469a5" (UID: "92906908-16f4-45e4-8177-1b53992469a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:31.154984 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.154929    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92906908-16f4-45e4-8177-1b53992469a5-kube-api-access-8txzf" pod "92906908-16f4-45e4-8177-1b53992469a5" (UID: "92906908-16f4-45e4-8177-1b53992469a5"). InnerVolumeSpecName "kube-api-access-8txzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:31.155164 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.155118    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-dshm" pod "92906908-16f4-45e4-8177-1b53992469a5" (UID: "92906908-16f4-45e4-8177-1b53992469a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:31.169713 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.168625    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92906908-16f4-45e4-8177-1b53992469a5-tls-certs" pod "92906908-16f4-45e4-8177-1b53992469a5" (UID: "92906908-16f4-45e4-8177-1b53992469a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:38:31.253431 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.253395    4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-dshm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:31.253587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.253455    4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-tmp-dir\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:31.253587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.253473    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8txzf\" (UniqueName: \"kubernetes.io/projected/92906908-16f4-45e4-8177-1b53992469a5-kube-api-access-8txzf\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:31.253587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.253489    4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-model-cache\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:31.253587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.253502    4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92906908-16f4-45e4-8177-1b53992469a5-home\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:31.253587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.253515    4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92906908-16f4-45e4-8177-1b53992469a5-tls-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:31.676569 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.676535    4441 generic.go:356] "Generic (PLEG): container finished" podID="92906908-16f4-45e4-8177-1b53992469a5" containerID="bd04ecaccc0e151cd7d0602eb86388fa082c9ee85a108c2ef6ac99d4b6a9cdda" exitCode=0
Oct 30 00:38:31.676814 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.676594    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"
Oct 30 00:38:31.676814 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.676628    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" event={"ID":"92906908-16f4-45e4-8177-1b53992469a5","Type":"ContainerDied","Data":"bd04ecaccc0e151cd7d0602eb86388fa082c9ee85a108c2ef6ac99d4b6a9cdda"}
Oct 30 00:38:31.676814 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.676680    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9" event={"ID":"92906908-16f4-45e4-8177-1b53992469a5","Type":"ContainerDied","Data":"c0869c73466b5c7285ca2af693d7b1c5073f0df0ec56ccffea9aa906120cfea0"}
Oct 30 00:38:31.676814 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.676704    4441 scope.go:122] "RemoveContainer" containerID="bd04ecaccc0e151cd7d0602eb86388fa082c9ee85a108c2ef6ac99d4b6a9cdda"
Oct 30 00:38:31.687366 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.687340    4441 scope.go:122] "RemoveContainer" containerID="bd04ecaccc0e151cd7d0602eb86388fa082c9ee85a108c2ef6ac99d4b6a9cdda"
Oct 30 00:38:31.688052 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:38:31.688011    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd04ecaccc0e151cd7d0602eb86388fa082c9ee85a108c2ef6ac99d4b6a9cdda\": container with ID starting with bd04ecaccc0e151cd7d0602eb86388fa082c9ee85a108c2ef6ac99d4b6a9cdda not found: ID does not exist" containerID="bd04ecaccc0e151cd7d0602eb86388fa082c9ee85a108c2ef6ac99d4b6a9cdda"
Oct 30 00:38:31.688274 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.688066    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd04ecaccc0e151cd7d0602eb86388fa082c9ee85a108c2ef6ac99d4b6a9cdda"} err="failed to get container status \"bd04ecaccc0e151cd7d0602eb86388fa082c9ee85a108c2ef6ac99d4b6a9cdda\": rpc error: code = NotFound desc = could not find container \"bd04ecaccc0e151cd7d0602eb86388fa082c9ee85a108c2ef6ac99d4b6a9cdda\": container with ID starting with bd04ecaccc0e151cd7d0602eb86388fa082c9ee85a108c2ef6ac99d4b6a9cdda not found: ID does not exist"
Oct 30 00:38:31.728565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.728530    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"]
Oct 30 00:38:31.739506 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:31.739474    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/simulator-w4-02665b-kserve-d4fcc57cb-kn8d9"]
Oct 30 00:38:32.609332 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:32.609301    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92906908-16f4-45e4-8177-1b53992469a5" path="/var/lib/kubelet/pods/92906908-16f4-45e4-8177-1b53992469a5/volumes"
Oct 30 00:38:33.902226 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:38:33.902178    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:38:34.284675 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:34.284634    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"]
Oct 30 00:38:35.937147 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:35.936936    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:38:36.158272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:36.158227    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm" podUID="759af21c-c080-4627-8863-b3f419cd4420" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:38:38.107477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.101468    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"]
Oct 30 00:38:38.107477 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.106537    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t" podUID="abdcfa70-8788-4f09-b420-7a1aa2e03314" containerName="maas-api" containerID="cri-o://a4b86677ef40e887131d7e29d98c7f1c24df71ba53218af4a5b67ae2f12693b5" gracePeriod=30
Oct 30 00:38:38.557061 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.552115    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:38:38.704413 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.704381    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abdcfa70-8788-4f09-b420-7a1aa2e03314-kube-api-access-gzqxm\" (UniqueName: \"kubernetes.io/projected/abdcfa70-8788-4f09-b420-7a1aa2e03314-kube-api-access-gzqxm\") pod \"abdcfa70-8788-4f09-b420-7a1aa2e03314\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") "
Oct 30 00:38:38.710254 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.709170    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-maas-api-tls\") pod \"abdcfa70-8788-4f09-b420-7a1aa2e03314\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") "
Oct 30 00:38:38.710254 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.709269    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abdcfa70-8788-4f09-b420-7a1aa2e03314-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdcfa70-8788-4f09-b420-7a1aa2e03314-openshift-service-ca\") pod \"abdcfa70-8788-4f09-b420-7a1aa2e03314\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") "
Oct 30 00:38:38.710254 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.709333    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-metrics-certs\") pod \"abdcfa70-8788-4f09-b420-7a1aa2e03314\" (UID: \"abdcfa70-8788-4f09-b420-7a1aa2e03314\") "
Oct 30 00:38:38.712285 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.712252    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abdcfa70-8788-4f09-b420-7a1aa2e03314-openshift-service-ca" pod "abdcfa70-8788-4f09-b420-7a1aa2e03314" (UID: "abdcfa70-8788-4f09-b420-7a1aa2e03314"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:38:38.713541 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.713511    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abdcfa70-8788-4f09-b420-7a1aa2e03314-kube-api-access-gzqxm" pod "abdcfa70-8788-4f09-b420-7a1aa2e03314" (UID: "abdcfa70-8788-4f09-b420-7a1aa2e03314"). InnerVolumeSpecName "kube-api-access-gzqxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:38.714453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.714429    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-metrics-certs" pod "abdcfa70-8788-4f09-b420-7a1aa2e03314" (UID: "abdcfa70-8788-4f09-b420-7a1aa2e03314"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:38:38.714665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.714642    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-maas-api-tls" pod "abdcfa70-8788-4f09-b420-7a1aa2e03314" (UID: "abdcfa70-8788-4f09-b420-7a1aa2e03314"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:38:38.756410 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.756379    4441 generic.go:356] "Generic (PLEG): container finished" podID="abdcfa70-8788-4f09-b420-7a1aa2e03314" containerID="a4b86677ef40e887131d7e29d98c7f1c24df71ba53218af4a5b67ae2f12693b5" exitCode=0
Oct 30 00:38:38.756530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.756426    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"
Oct 30 00:38:38.756530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.756455    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t" event={"ID":"abdcfa70-8788-4f09-b420-7a1aa2e03314","Type":"ContainerDied","Data":"a4b86677ef40e887131d7e29d98c7f1c24df71ba53218af4a5b67ae2f12693b5"}
Oct 30 00:38:38.756530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.756513    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t" event={"ID":"abdcfa70-8788-4f09-b420-7a1aa2e03314","Type":"ContainerDied","Data":"e694dba02bb6e92a3512ce821b57547525a4f2f0be63c1dabd65a2397c4245cb"}
Oct 30 00:38:38.756698 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.756541    4441 scope.go:122] "RemoveContainer" containerID="a4b86677ef40e887131d7e29d98c7f1c24df71ba53218af4a5b67ae2f12693b5"
Oct 30 00:38:38.768530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.768512    4441 scope.go:122] "RemoveContainer" containerID="a4b86677ef40e887131d7e29d98c7f1c24df71ba53218af4a5b67ae2f12693b5"
Oct 30 00:38:38.768827 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:38:38.768808    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b86677ef40e887131d7e29d98c7f1c24df71ba53218af4a5b67ae2f12693b5\": container with ID starting with a4b86677ef40e887131d7e29d98c7f1c24df71ba53218af4a5b67ae2f12693b5 not found: ID does not exist" containerID="a4b86677ef40e887131d7e29d98c7f1c24df71ba53218af4a5b67ae2f12693b5"
Oct 30 00:38:38.768903 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.768846    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b86677ef40e887131d7e29d98c7f1c24df71ba53218af4a5b67ae2f12693b5"} err="failed to get container status \"a4b86677ef40e887131d7e29d98c7f1c24df71ba53218af4a5b67ae2f12693b5\": rpc error: code = NotFound desc = could not find container \"a4b86677ef40e887131d7e29d98c7f1c24df71ba53218af4a5b67ae2f12693b5\": container with ID starting with a4b86677ef40e887131d7e29d98c7f1c24df71ba53218af4a5b67ae2f12693b5 not found: ID does not exist"
Oct 30 00:38:38.809555 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.809533    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"]
Oct 30 00:38:38.813940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.813921    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:38.814075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.813945    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdcfa70-8788-4f09-b420-7a1aa2e03314-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:38.814075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.813958    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abdcfa70-8788-4f09-b420-7a1aa2e03314-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:38.814075 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.813968    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzqxm\" (UniqueName: \"kubernetes.io/projected/abdcfa70-8788-4f09-b420-7a1aa2e03314-kube-api-access-gzqxm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:38.814887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:38.814867    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-02665b-6cc76cf879-db97t"]
Oct 30 00:38:40.609695 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:40.609661    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abdcfa70-8788-4f09-b420-7a1aa2e03314" path="/var/lib/kubelet/pods/abdcfa70-8788-4f09-b420-7a1aa2e03314/volumes"
Oct 30 00:38:45.960000 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:45.959965    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"]
Oct 30 00:38:45.960507 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:45.960247    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerName="istio-proxy" containerID="cri-o://425187aad361de855323a9f3b522ef3d57b0e1233ac6c4b1653b706fe6ae8a9c" gracePeriod=30
Oct 30 00:38:46.173284 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:46.171247    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm" podUID="759af21c-c080-4627-8863-b3f419cd4420" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:38:46.794368 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:46.794344    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"
Oct 30 00:38:46.820347 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:46.820320    4441 generic.go:356] "Generic (PLEG): container finished" podID="759af21c-c080-4627-8863-b3f419cd4420" containerID="8a85a64d901ba20f90974954493e7cb45e4068bbc1fdbb6786a7874562504a2d" exitCode=0
Oct 30 00:38:46.820492 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:46.820357    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"
Oct 30 00:38:46.820492 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:46.820394    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm" event={"ID":"759af21c-c080-4627-8863-b3f419cd4420","Type":"ContainerDied","Data":"8a85a64d901ba20f90974954493e7cb45e4068bbc1fdbb6786a7874562504a2d"}
Oct 30 00:38:46.820642 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:46.820495    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm" event={"ID":"759af21c-c080-4627-8863-b3f419cd4420","Type":"ContainerDied","Data":"ce6d4495c4cf1d77bba5becf2027a8b4539709e8096552f77646f5547cdfabf8"}
Oct 30 00:38:46.820642 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:46.820530    4441 scope.go:122] "RemoveContainer" containerID="8a85a64d901ba20f90974954493e7cb45e4068bbc1fdbb6786a7874562504a2d"
Oct 30 00:38:46.831851 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:46.831836    4441 scope.go:122] "RemoveContainer" containerID="8a85a64d901ba20f90974954493e7cb45e4068bbc1fdbb6786a7874562504a2d"
Oct 30 00:38:46.832130 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:38:46.832108    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a85a64d901ba20f90974954493e7cb45e4068bbc1fdbb6786a7874562504a2d\": container with ID starting with 8a85a64d901ba20f90974954493e7cb45e4068bbc1fdbb6786a7874562504a2d not found: ID does not exist" containerID="8a85a64d901ba20f90974954493e7cb45e4068bbc1fdbb6786a7874562504a2d"
Oct 30 00:38:46.832217 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:46.832140    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a85a64d901ba20f90974954493e7cb45e4068bbc1fdbb6786a7874562504a2d"} err="failed to get container status \"8a85a64d901ba20f90974954493e7cb45e4068bbc1fdbb6786a7874562504a2d\": rpc error: code = NotFound desc = could not find container \"8a85a64d901ba20f90974954493e7cb45e4068bbc1fdbb6786a7874562504a2d\": container with ID starting with 8a85a64d901ba20f90974954493e7cb45e4068bbc1fdbb6786a7874562504a2d not found: ID does not exist"
Oct 30 00:38:46.955722 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:46.955692    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/759af21c-c080-4627-8863-b3f419cd4420-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759af21c-c080-4627-8863-b3f419cd4420-plugins-config-volume\") pod \"759af21c-c080-4627-8863-b3f419cd4420\" (UID: \"759af21c-c080-4627-8863-b3f419cd4420\") "
Oct 30 00:38:46.955930 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:46.955917    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759af21c-c080-4627-8863-b3f419cd4420-kube-api-access-7rw7p\" (UniqueName: \"kubernetes.io/projected/759af21c-c080-4627-8863-b3f419cd4420-kube-api-access-7rw7p\") pod \"759af21c-c080-4627-8863-b3f419cd4420\" (UID: \"759af21c-c080-4627-8863-b3f419cd4420\") "
Oct 30 00:38:46.956172 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:46.956148    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759af21c-c080-4627-8863-b3f419cd4420-plugins-config-volume" pod "759af21c-c080-4627-8863-b3f419cd4420" (UID: "759af21c-c080-4627-8863-b3f419cd4420"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:38:46.957914 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:46.957891    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759af21c-c080-4627-8863-b3f419cd4420-kube-api-access-7rw7p" pod "759af21c-c080-4627-8863-b3f419cd4420" (UID: "759af21c-c080-4627-8863-b3f419cd4420"). InnerVolumeSpecName "kube-api-access-7rw7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:47.057356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:47.057276    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rw7p\" (UniqueName: \"kubernetes.io/projected/759af21c-c080-4627-8863-b3f419cd4420-kube-api-access-7rw7p\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:47.057356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:47.057303    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759af21c-c080-4627-8863-b3f419cd4420-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:47.166839 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:47.166804    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"]
Oct 30 00:38:47.177342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:47.177313    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-02665b-5477b77f-d69xm"]
Oct 30 00:38:48.610497 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:48.610461    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="759af21c-c080-4627-8863-b3f419cd4420" path="/var/lib/kubelet/pods/759af21c-c080-4627-8863-b3f419cd4420/volumes"
Oct 30 00:38:49.305097 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.305012    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch" podUID="bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4" containerName="main" containerID="cri-o://90687fe38514e0b500003a7d2b4d6b7d9210d6ed5906a4a4344e6b64332a1a7a" gracePeriod=45
Oct 30 00:38:49.458313 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.458266    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch" podUID="bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.171:8000/ready\": dial tcp 10.133.0.171:8000: connect: connection refused"
Oct 30 00:38:49.593896 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.593868    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:38:49.606157 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.606128    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tls-certs\" (UniqueName: \"kubernetes.io/secret/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tls-certs\") pod \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") "
Oct 30 00:38:49.606317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.606177    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tmp-dir\") pod \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") "
Oct 30 00:38:49.606317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.606228    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-kube-api-access-5rwzc\" (UniqueName: \"kubernetes.io/projected/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-kube-api-access-5rwzc\") pod \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") "
Oct 30 00:38:49.606317 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.606295    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-home\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-home\") pod \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") "
Oct 30 00:38:49.606492 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.606362    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-model-cache\") pod \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") "
Oct 30 00:38:49.606492 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.606412    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-dshm\") pod \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\" (UID: \"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4\") "
Oct 30 00:38:49.606599 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.606500    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tmp-dir" pod "bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4" (UID: "bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:49.606683 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.606626    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-home" pod "bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4" (UID: "bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:49.606747 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.606686    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-model-cache" pod "bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4" (UID: "bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:49.606942 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.606916    4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-home\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:49.607092 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.606945    4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-model-cache\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:49.607092 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.606960    4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tmp-dir\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:49.608573 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.608546    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tls-certs" pod "bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4" (UID: "bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:38:49.608573 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.608554    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-dshm" pod "bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4" (UID: "bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:49.608742 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.608600    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-kube-api-access-5rwzc" pod "bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4" (UID: "bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4"). InnerVolumeSpecName "kube-api-access-5rwzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:49.707924 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.707882    4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-dshm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:49.707924 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.707917    4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-tls-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:49.707924 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.707930    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rwzc\" (UniqueName: \"kubernetes.io/projected/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4-kube-api-access-5rwzc\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:49.908458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.908369    4441 generic.go:356] "Generic (PLEG): container finished" podID="bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4" containerID="90687fe38514e0b500003a7d2b4d6b7d9210d6ed5906a4a4344e6b64332a1a7a" exitCode=0
Oct 30 00:38:49.908458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.908413    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"
Oct 30 00:38:49.908458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.908447    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch" event={"ID":"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4","Type":"ContainerDied","Data":"90687fe38514e0b500003a7d2b4d6b7d9210d6ed5906a4a4344e6b64332a1a7a"}
Oct 30 00:38:49.908739 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.908500    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch" event={"ID":"bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4","Type":"ContainerDied","Data":"4b8d6c189712eb110aff7753f60daf60f68ede4beb75d0777923f758544f82a0"}
Oct 30 00:38:49.908739 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.908518    4441 scope.go:122] "RemoveContainer" containerID="90687fe38514e0b500003a7d2b4d6b7d9210d6ed5906a4a4344e6b64332a1a7a"
Oct 30 00:38:49.919197 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.919174    4441 scope.go:122] "RemoveContainer" containerID="90687fe38514e0b500003a7d2b4d6b7d9210d6ed5906a4a4344e6b64332a1a7a"
Oct 30 00:38:49.919458 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:38:49.919440    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90687fe38514e0b500003a7d2b4d6b7d9210d6ed5906a4a4344e6b64332a1a7a\": container with ID starting with 90687fe38514e0b500003a7d2b4d6b7d9210d6ed5906a4a4344e6b64332a1a7a not found: ID does not exist" containerID="90687fe38514e0b500003a7d2b4d6b7d9210d6ed5906a4a4344e6b64332a1a7a"
Oct 30 00:38:49.919505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.919465    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90687fe38514e0b500003a7d2b4d6b7d9210d6ed5906a4a4344e6b64332a1a7a"} err="failed to get container status \"90687fe38514e0b500003a7d2b4d6b7d9210d6ed5906a4a4344e6b64332a1a7a\": rpc error: code = NotFound desc = could not find container \"90687fe38514e0b500003a7d2b4d6b7d9210d6ed5906a4a4344e6b64332a1a7a\": container with ID starting with 90687fe38514e0b500003a7d2b4d6b7d9210d6ed5906a4a4344e6b64332a1a7a not found: ID does not exist"
Oct 30 00:38:49.960036 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.959999    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"]
Oct 30 00:38:49.969857 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:49.969830    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-f1268c/auth-test-model-f1268c-kserve-86f445cf7c-kfpch"]
Oct 30 00:38:50.609259 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.609230    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4" path="/var/lib/kubelet/pods/bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4/volumes"
Oct 30 00:38:50.754576 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.754546    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"]
Oct 30 00:38:50.755082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755066    4441 cpu_manager.go:414] "containerMap: removing container" podUID="bc0d0e19-9703-41aa-8a3c-4e0a85057b94" containerName="maas-api"
Oct 30 00:38:50.755082 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755082    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc0d0e19-9703-41aa-8a3c-4e0a85057b94" containerName="maas-api"
Oct 30 00:38:50.755167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755097    4441 cpu_manager.go:414] "containerMap: removing container" podUID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" containerName="istio-proxy"
Oct 30 00:38:50.755167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755103    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" containerName="istio-proxy"
Oct 30 00:38:50.755167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755115    4441 cpu_manager.go:414] "containerMap: removing container" podUID="bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4" containerName="main"
Oct 30 00:38:50.755167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755119    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4" containerName="main"
Oct 30 00:38:50.755167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755128    4441 cpu_manager.go:414] "containerMap: removing container" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy"
Oct 30 00:38:50.755167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755133    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy"
Oct 30 00:38:50.755167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755140    4441 cpu_manager.go:414] "containerMap: removing container" podUID="0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c" containerName="payload-pre-processing"
Oct 30 00:38:50.755167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755145    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c" containerName="payload-pre-processing"
Oct 30 00:38:50.755167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755153    4441 cpu_manager.go:414] "containerMap: removing container" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main"
Oct 30 00:38:50.755167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755159    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main"
Oct 30 00:38:50.755167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755169    4441 cpu_manager.go:414] "containerMap: removing container" podUID="4e076602-4664-4841-8894-75908287152d" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755174    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e076602-4664-4841-8894-75908287152d" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755214    4441 cpu_manager.go:414] "containerMap: removing container" podUID="759af21c-c080-4627-8863-b3f419cd4420" containerName="payload-pre-processing"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755222    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759af21c-c080-4627-8863-b3f419cd4420" containerName="payload-pre-processing"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755234    4441 cpu_manager.go:414] "containerMap: removing container" podUID="58e7b78b-01af-4965-a2ee-0f73c0bf2e58" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755239    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e7b78b-01af-4965-a2ee-0f73c0bf2e58" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755251    4441 cpu_manager.go:414] "containerMap: removing container" podUID="02017bad-fe6f-415d-a24e-4b6b26a2a99a" containerName="payload-processing"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755256    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02017bad-fe6f-415d-a24e-4b6b26a2a99a" containerName="payload-processing"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755264    4441 cpu_manager.go:414] "containerMap: removing container" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerName="istio-proxy"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755268    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerName="istio-proxy"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755277    4441 cpu_manager.go:414] "containerMap: removing container" podUID="2f7ee5a9-ec28-41f7-8dcf-0a88595c3400" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755282    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7ee5a9-ec28-41f7-8dcf-0a88595c3400" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755290    4441 cpu_manager.go:414] "containerMap: removing container" podUID="2ae884f6-aa1a-4aca-addf-a0c37c26aa80" containerName="istio-proxy"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755294    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae884f6-aa1a-4aca-addf-a0c37c26aa80" containerName="istio-proxy"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755301    4441 cpu_manager.go:414] "containerMap: removing container" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerName="istio-proxy"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755306    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerName="istio-proxy"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755315    4441 cpu_manager.go:414] "containerMap: removing container" podUID="abdcfa70-8788-4f09-b420-7a1aa2e03314" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755319    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abdcfa70-8788-4f09-b420-7a1aa2e03314" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755326    4441 cpu_manager.go:414] "containerMap: removing container" podUID="b4977042-bf8c-4bc6-8462-7f84acf9e5a2" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755331    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4977042-bf8c-4bc6-8462-7f84acf9e5a2" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755416    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="92906908-16f4-45e4-8177-1b53992469a5" containerName="main"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755423    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e076602-4664-4841-8894-75908287152d" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755432    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="759af21c-c080-4627-8863-b3f419cd4420" containerName="payload-pre-processing"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755439    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e7b78b-01af-4965-a2ee-0f73c0bf2e58" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755447    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="02017bad-fe6f-415d-a24e-4b6b26a2a99a" containerName="payload-processing"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755453    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755459    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerName="istio-proxy"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755465    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7ee5a9-ec28-41f7-8dcf-0a88595c3400" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755471    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae884f6-aa1a-4aca-addf-a0c37c26aa80" containerName="istio-proxy"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755476    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66cdbd4-e6a1-44e2-b459-c1502f57225a" containerName="istio-proxy"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755482    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="abdcfa70-8788-4f09-b420-7a1aa2e03314" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755487    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4977042-bf8c-4bc6-8462-7f84acf9e5a2" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755493    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc0d0e19-9703-41aa-8a3c-4e0a85057b94" containerName="maas-api"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755498    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c8a2b4-c0aa-41f2-a0af-80de573b6fb5" containerName="istio-proxy"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755503    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbaefb10-ab6e-4c79-b60d-f6c7e32bddf4" containerName="main"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755508    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy"
Oct 30 00:38:50.755502 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.755514    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7539ff-c7d9-475c-80a4-6cdbd29f4c6c" containerName="payload-pre-processing"
Oct 30 00:38:50.760272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.760257    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:50.790008 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.789972    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"]
Oct 30 00:38:50.815727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.815703    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-d96837-serving-cert\""
Oct 30 00:38:50.821041 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.820998    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2143da-e83f-4fad-a3fe-7509571c484e-openshift-service-ca\") pod \"maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:50.821195 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.821078    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gvn\" (UniqueName: \"kubernetes.io/projected/0e2143da-e83f-4fad-a3fe-7509571c484e-kube-api-access-22gvn\") pod \"maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:50.821256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.821207    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-metrics-certs\") pod \"maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:50.821256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.821244    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-maas-api-tls\") pod \"maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:50.922633 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.922536    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2143da-e83f-4fad-a3fe-7509571c484e-openshift-service-ca\") pod \"maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:50.922790 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.922692    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gvn\" (UniqueName: \"kubernetes.io/projected/0e2143da-e83f-4fad-a3fe-7509571c484e-kube-api-access-22gvn\") pod \"maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:50.922843 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.922831    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-metrics-certs\") pod \"maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:50.922888 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.922873    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-maas-api-tls\") pod \"maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:50.923359 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.923326    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2143da-e83f-4fad-a3fe-7509571c484e-openshift-service-ca\") pod \"maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:50.925505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.925479    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-metrics-certs\") pod \"maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:50.925779 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.925760    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-maas-api-tls\") pod \"maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:50.945350 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:50.945315    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gvn\" (UniqueName: \"kubernetes.io/projected/0e2143da-e83f-4fad-a3fe-7509571c484e-kube-api-access-22gvn\") pod \"maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:51.078781 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.078313    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:51.292624 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.292600    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:38:51.340971 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.340938    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-credential-socket\") pod \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") "
Oct 30 00:38:51.341178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.340996    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-token\" (UniqueName: \"kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-token\") pod \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") "
Oct 30 00:38:51.341178 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.341116    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-data\") pod \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") "
Oct 30 00:38:51.341316 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.341182    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-kube-api-access-xljcd\" (UniqueName: \"kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-kube-api-access-xljcd\") pod \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") "
Oct 30 00:38:51.341316 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.341264    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-certs\") pod \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") "
Oct 30 00:38:51.341316 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.341294    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-socket\") pod \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") "
Oct 30 00:38:51.341490 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.341355    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-envoy\") pod \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") "
Oct 30 00:38:51.341490 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.341429    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istiod-ca-cert\") pod \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") "
Oct 30 00:38:51.346763 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.341489    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-podinfo\") pod \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\" (UID: \"377ffe8e-cc09-4c8b-a25d-c2e486236de5\") "
Oct 30 00:38:51.346763 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.345633    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-credential-socket" pod "377ffe8e-cc09-4c8b-a25d-c2e486236de5" (UID: "377ffe8e-cc09-4c8b-a25d-c2e486236de5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:51.346763 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.345942    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-socket" pod "377ffe8e-cc09-4c8b-a25d-c2e486236de5" (UID: "377ffe8e-cc09-4c8b-a25d-c2e486236de5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:51.346763 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.346209    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-certs" pod "377ffe8e-cc09-4c8b-a25d-c2e486236de5" (UID: "377ffe8e-cc09-4c8b-a25d-c2e486236de5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:51.347077 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.346937    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istiod-ca-cert" pod "377ffe8e-cc09-4c8b-a25d-c2e486236de5" (UID: "377ffe8e-cc09-4c8b-a25d-c2e486236de5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:38:51.348370 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.347510    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-data" pod "377ffe8e-cc09-4c8b-a25d-c2e486236de5" (UID: "377ffe8e-cc09-4c8b-a25d-c2e486236de5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:51.350705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.350663    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-envoy" pod "377ffe8e-cc09-4c8b-a25d-c2e486236de5" (UID: "377ffe8e-cc09-4c8b-a25d-c2e486236de5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:51.353223 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.353195    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-token" pod "377ffe8e-cc09-4c8b-a25d-c2e486236de5" (UID: "377ffe8e-cc09-4c8b-a25d-c2e486236de5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:51.354213 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.354191    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-kube-api-access-xljcd" pod "377ffe8e-cc09-4c8b-a25d-c2e486236de5" (UID: "377ffe8e-cc09-4c8b-a25d-c2e486236de5"). InnerVolumeSpecName "kube-api-access-xljcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:51.354306 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.354289    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-podinfo" pod "377ffe8e-cc09-4c8b-a25d-c2e486236de5" (UID: "377ffe8e-cc09-4c8b-a25d-c2e486236de5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:38:51.404140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.404114    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"]
Oct 30 00:38:51.405447 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:38:51.405422    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2143da_e83f_4fad_a3fe_7509571c484e.slice/crio-2c670981ea1748b0da8c22ff8a62809ad2aba3f183f48a3e5a765bcdc9804156 WatchSource:0}: Error finding container 2c670981ea1748b0da8c22ff8a62809ad2aba3f183f48a3e5a765bcdc9804156: Status 404 returned error can't find the container with id 2c670981ea1748b0da8c22ff8a62809ad2aba3f183f48a3e5a765bcdc9804156
Oct 30 00:38:51.406801 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.406785    4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:38:51.443162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.443137    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:51.443162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.443162    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:51.443342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.443172    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:51.443342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.443183    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xljcd\" (UniqueName: \"kubernetes.io/projected/377ffe8e-cc09-4c8b-a25d-c2e486236de5-kube-api-access-xljcd\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:51.443342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.443191    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:51.443342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.443199    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:51.443342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.443207    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:51.443342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.443216    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:51.443342 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.443224    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377ffe8e-cc09-4c8b-a25d-c2e486236de5-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:38:51.923465 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.923349    4441 generic.go:356] "Generic (PLEG): container finished" podID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerID="425187aad361de855323a9f3b522ef3d57b0e1233ac6c4b1653b706fe6ae8a9c" exitCode=0
Oct 30 00:38:51.924653 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.923475    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"
Oct 30 00:38:51.924653 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.923654    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" event={"ID":"377ffe8e-cc09-4c8b-a25d-c2e486236de5","Type":"ContainerDied","Data":"425187aad361de855323a9f3b522ef3d57b0e1233ac6c4b1653b706fe6ae8a9c"}
Oct 30 00:38:51.924653 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.923711    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln" event={"ID":"377ffe8e-cc09-4c8b-a25d-c2e486236de5","Type":"ContainerDied","Data":"1c779cf1487774bed98756bc1b875a22a7016ebd4535365bbf816a25a1e7acc0"}
Oct 30 00:38:51.924653 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.923739    4441 scope.go:122] "RemoveContainer" containerID="425187aad361de855323a9f3b522ef3d57b0e1233ac6c4b1653b706fe6ae8a9c"
Oct 30 00:38:51.928067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.928002    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs" event={"ID":"0e2143da-e83f-4fad-a3fe-7509571c484e","Type":"ContainerStarted","Data":"2c670981ea1748b0da8c22ff8a62809ad2aba3f183f48a3e5a765bcdc9804156"}
Oct 30 00:38:51.992931 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.992904    4441 scope.go:122] "RemoveContainer" containerID="a25a1c66f47386d6aa90a9c7ba8dd7a6bbcb697db8ec209a467a7cc59eea78f0"
Oct 30 00:38:51.996288 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:51.996266    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"]
Oct 30 00:38:52.004478 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:52.004388    4441 scope.go:122] "RemoveContainer" containerID="425187aad361de855323a9f3b522ef3d57b0e1233ac6c4b1653b706fe6ae8a9c"
Oct 30 00:38:52.004744 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:38:52.004716    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425187aad361de855323a9f3b522ef3d57b0e1233ac6c4b1653b706fe6ae8a9c\": container with ID starting with 425187aad361de855323a9f3b522ef3d57b0e1233ac6c4b1653b706fe6ae8a9c not found: ID does not exist" containerID="425187aad361de855323a9f3b522ef3d57b0e1233ac6c4b1653b706fe6ae8a9c"
Oct 30 00:38:52.005325 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:52.004752    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425187aad361de855323a9f3b522ef3d57b0e1233ac6c4b1653b706fe6ae8a9c"} err="failed to get container status \"425187aad361de855323a9f3b522ef3d57b0e1233ac6c4b1653b706fe6ae8a9c\": rpc error: code = NotFound desc = could not find container \"425187aad361de855323a9f3b522ef3d57b0e1233ac6c4b1653b706fe6ae8a9c\": container with ID starting with 425187aad361de855323a9f3b522ef3d57b0e1233ac6c4b1653b706fe6ae8a9c not found: ID does not exist"
Oct 30 00:38:52.005325 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:52.004774    4441 scope.go:122] "RemoveContainer" containerID="a25a1c66f47386d6aa90a9c7ba8dd7a6bbcb697db8ec209a467a7cc59eea78f0"
Oct 30 00:38:52.005325 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:38:52.005067    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25a1c66f47386d6aa90a9c7ba8dd7a6bbcb697db8ec209a467a7cc59eea78f0\": container with ID starting with a25a1c66f47386d6aa90a9c7ba8dd7a6bbcb697db8ec209a467a7cc59eea78f0 not found: ID does not exist" containerID="a25a1c66f47386d6aa90a9c7ba8dd7a6bbcb697db8ec209a467a7cc59eea78f0"
Oct 30 00:38:52.005325 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:52.005095    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25a1c66f47386d6aa90a9c7ba8dd7a6bbcb697db8ec209a467a7cc59eea78f0"} err="failed to get container status \"a25a1c66f47386d6aa90a9c7ba8dd7a6bbcb697db8ec209a467a7cc59eea78f0\": rpc error: code = NotFound desc = could not find container \"a25a1c66f47386d6aa90a9c7ba8dd7a6bbcb697db8ec209a467a7cc59eea78f0\": container with ID starting with a25a1c66f47386d6aa90a9c7ba8dd7a6bbcb697db8ec209a467a7cc59eea78f0 not found: ID does not exist"
Oct 30 00:38:52.007625 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:52.007598    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kvgln"]
Oct 30 00:38:52.609343 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:52.609310    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" path="/var/lib/kubelet/pods/377ffe8e-cc09-4c8b-a25d-c2e486236de5/volumes"
Oct 30 00:38:52.951649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:52.951551    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs" event={"ID":"0e2143da-e83f-4fad-a3fe-7509571c484e","Type":"ContainerStarted","Data":"8b0a7c572b9d5994735b003b7aae41399a2b74bb74c374715a8644d2d3a4eea8"}
Oct 30 00:38:52.952153 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:52.951736    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:38:56.198117 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:56.192979    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs" podStartSLOduration=5.730440505 podStartE2EDuration="6.19295653s" totalImagesPullingTime="462.516025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:53.020339304 +0000 UTC m=+2042.545633181" watchObservedRunningTime="2026-10-30 00:38:56.19295653 +0000 UTC m=+2045.718250408"
Oct 30 00:38:56.198117 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:56.195227    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"]
Oct 30 00:38:56.198117 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:56.195493    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" containerName="istio-proxy" containerID="cri-o://cefadf30dfd6be3ea3fb5228ae0ff98396fb9437ddeadf9fbcd626e8f4ddcfe6" gracePeriod=30
Oct 30 00:38:58.965152 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:38:58.965120    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:39:01.474120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.474055    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:39:01.581348 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.581314    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c0fec19-a16c-40a1-8840-66022d000e75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0fec19-a16c-40a1-8840-66022d000e75-istio-podinfo\") pod \"5c0fec19-a16c-40a1-8840-66022d000e75\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") "
Oct 30 00:39:01.581528 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.581374    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-data\") pod \"5c0fec19-a16c-40a1-8840-66022d000e75\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") "
Oct 30 00:39:01.581528 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.581402    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-kube-api-access-f55p9\" (UniqueName: \"kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-kube-api-access-f55p9\") pod \"5c0fec19-a16c-40a1-8840-66022d000e75\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") "
Oct 30 00:39:01.581667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.581536    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-istio-token\" (UniqueName: \"kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-istio-token\") pod \"5c0fec19-a16c-40a1-8840-66022d000e75\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") "
Oct 30 00:39:01.581667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.581591    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-certs\") pod \"5c0fec19-a16c-40a1-8840-66022d000e75\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") "
Oct 30 00:39:01.581667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.581658    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-credential-socket\") pod \"5c0fec19-a16c-40a1-8840-66022d000e75\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") "
Oct 30 00:39:01.581835 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.581687    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c0fec19-a16c-40a1-8840-66022d000e75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0fec19-a16c-40a1-8840-66022d000e75-istiod-ca-cert\") pod \"5c0fec19-a16c-40a1-8840-66022d000e75\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") "
Oct 30 00:39:01.581835 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.581701    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-data" pod "5c0fec19-a16c-40a1-8840-66022d000e75" (UID: "5c0fec19-a16c-40a1-8840-66022d000e75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:01.581835 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.581725    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-socket\") pod \"5c0fec19-a16c-40a1-8840-66022d000e75\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") "
Oct 30 00:39:01.581835 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.581773    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-envoy\") pod \"5c0fec19-a16c-40a1-8840-66022d000e75\" (UID: \"5c0fec19-a16c-40a1-8840-66022d000e75\") "
Oct 30 00:39:01.582065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.581913    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-certs" pod "5c0fec19-a16c-40a1-8840-66022d000e75" (UID: "5c0fec19-a16c-40a1-8840-66022d000e75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:01.582408 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.582134    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-socket" pod "5c0fec19-a16c-40a1-8840-66022d000e75" (UID: "5c0fec19-a16c-40a1-8840-66022d000e75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:01.582408 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.582168    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-credential-socket" pod "5c0fec19-a16c-40a1-8840-66022d000e75" (UID: "5c0fec19-a16c-40a1-8840-66022d000e75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:01.582408 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.582186    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0fec19-a16c-40a1-8840-66022d000e75-istiod-ca-cert" pod "5c0fec19-a16c-40a1-8840-66022d000e75" (UID: "5c0fec19-a16c-40a1-8840-66022d000e75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:39:01.582639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.582413    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:01.582639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.582432    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:01.582639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.582445    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0fec19-a16c-40a1-8840-66022d000e75-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:01.582639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.582459    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:01.582639 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.582472    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:01.584557 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.584524    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-istio-token" pod "5c0fec19-a16c-40a1-8840-66022d000e75" (UID: "5c0fec19-a16c-40a1-8840-66022d000e75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:01.584790 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.584764    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c0fec19-a16c-40a1-8840-66022d000e75-istio-podinfo" pod "5c0fec19-a16c-40a1-8840-66022d000e75" (UID: "5c0fec19-a16c-40a1-8840-66022d000e75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:39:01.584904 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.584792    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-envoy" pod "5c0fec19-a16c-40a1-8840-66022d000e75" (UID: "5c0fec19-a16c-40a1-8840-66022d000e75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:01.584904 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.584798    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-kube-api-access-f55p9" pod "5c0fec19-a16c-40a1-8840-66022d000e75" (UID: "5c0fec19-a16c-40a1-8840-66022d000e75"). InnerVolumeSpecName "kube-api-access-f55p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:01.684442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.684413    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0fec19-a16c-40a1-8840-66022d000e75-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:01.684442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.684438    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0fec19-a16c-40a1-8840-66022d000e75-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:01.684442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.684448    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55p9\" (UniqueName: \"kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-kube-api-access-f55p9\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:01.684754 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.684458    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0fec19-a16c-40a1-8840-66022d000e75-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:01.732404 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.732371    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"]
Oct 30 00:39:01.740209 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:01.740183    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"]
Oct 30 00:39:02.035958 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.035869    4441 generic.go:356] "Generic (PLEG): container finished" podID="5c0fec19-a16c-40a1-8840-66022d000e75" containerID="cefadf30dfd6be3ea3fb5228ae0ff98396fb9437ddeadf9fbcd626e8f4ddcfe6" exitCode=0
Oct 30 00:39:02.036122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.035951    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"
Oct 30 00:39:02.036122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.035966    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" event={"ID":"5c0fec19-a16c-40a1-8840-66022d000e75","Type":"ContainerDied","Data":"cefadf30dfd6be3ea3fb5228ae0ff98396fb9437ddeadf9fbcd626e8f4ddcfe6"}
Oct 30 00:39:02.036122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.036051    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt" event={"ID":"5c0fec19-a16c-40a1-8840-66022d000e75","Type":"ContainerDied","Data":"ec14f94b420c63e2cb805694d3f80ec9b686c4243187a0964b5609a66f2fde01"}
Oct 30 00:39:02.036122 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.036069    4441 scope.go:122] "RemoveContainer" containerID="cefadf30dfd6be3ea3fb5228ae0ff98396fb9437ddeadf9fbcd626e8f4ddcfe6"
Oct 30 00:39:02.046344 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.046320    4441 scope.go:122] "RemoveContainer" containerID="227177ae1363bdfb5b2019cfc6fc273f4814a4fa1a8ab60ed83cc318a187b931"
Oct 30 00:39:02.056618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.056598    4441 scope.go:122] "RemoveContainer" containerID="cefadf30dfd6be3ea3fb5228ae0ff98396fb9437ddeadf9fbcd626e8f4ddcfe6"
Oct 30 00:39:02.056868 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:02.056850    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefadf30dfd6be3ea3fb5228ae0ff98396fb9437ddeadf9fbcd626e8f4ddcfe6\": container with ID starting with cefadf30dfd6be3ea3fb5228ae0ff98396fb9437ddeadf9fbcd626e8f4ddcfe6 not found: ID does not exist" containerID="cefadf30dfd6be3ea3fb5228ae0ff98396fb9437ddeadf9fbcd626e8f4ddcfe6"
Oct 30 00:39:02.056919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.056875    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefadf30dfd6be3ea3fb5228ae0ff98396fb9437ddeadf9fbcd626e8f4ddcfe6"} err="failed to get container status \"cefadf30dfd6be3ea3fb5228ae0ff98396fb9437ddeadf9fbcd626e8f4ddcfe6\": rpc error: code = NotFound desc = could not find container \"cefadf30dfd6be3ea3fb5228ae0ff98396fb9437ddeadf9fbcd626e8f4ddcfe6\": container with ID starting with cefadf30dfd6be3ea3fb5228ae0ff98396fb9437ddeadf9fbcd626e8f4ddcfe6 not found: ID does not exist"
Oct 30 00:39:02.056919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.056899    4441 scope.go:122] "RemoveContainer" containerID="227177ae1363bdfb5b2019cfc6fc273f4814a4fa1a8ab60ed83cc318a187b931"
Oct 30 00:39:02.057201 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:02.057181    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227177ae1363bdfb5b2019cfc6fc273f4814a4fa1a8ab60ed83cc318a187b931\": container with ID starting with 227177ae1363bdfb5b2019cfc6fc273f4814a4fa1a8ab60ed83cc318a187b931 not found: ID does not exist" containerID="227177ae1363bdfb5b2019cfc6fc273f4814a4fa1a8ab60ed83cc318a187b931"
Oct 30 00:39:02.057250 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.057208    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227177ae1363bdfb5b2019cfc6fc273f4814a4fa1a8ab60ed83cc318a187b931"} err="failed to get container status \"227177ae1363bdfb5b2019cfc6fc273f4814a4fa1a8ab60ed83cc318a187b931\": rpc error: code = NotFound desc = could not find container \"227177ae1363bdfb5b2019cfc6fc273f4814a4fa1a8ab60ed83cc318a187b931\": container with ID starting with 227177ae1363bdfb5b2019cfc6fc273f4814a4fa1a8ab60ed83cc318a187b931 not found: ID does not exist"
Oct 30 00:39:02.093903 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.093874    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"]
Oct 30 00:39:02.097279 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.097248    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-415d24-openshift-default-67cf5cd765-628zt"]
Oct 30 00:39:02.609483 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.609452    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" path="/var/lib/kubelet/pods/5c0fec19-a16c-40a1-8840-66022d000e75/volumes"
Oct 30 00:39:02.836723 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.836689    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"]
Oct 30 00:39:02.836954 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:02.836912    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg" podUID="d59de270-9763-49fa-8efc-625d58575e48" containerName="istio-proxy" containerID="cri-o://35b3816932b30c2af46a88dc1ab8a87f7c6a5fd5d820114ca5daeb8377be4cdf" gracePeriod=30
Oct 30 00:39:03.902670 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:03.902636    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:39:05.332825 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:05.332787    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"]
Oct 30 00:39:05.347869 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:05.347838    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"]
Oct 30 00:39:07.458784 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:07.458740    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:39:08.080136 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.080097    4441 generic.go:356] "Generic (PLEG): container finished" podID="d59de270-9763-49fa-8efc-625d58575e48" containerID="35b3816932b30c2af46a88dc1ab8a87f7c6a5fd5d820114ca5daeb8377be4cdf" exitCode=0
Oct 30 00:39:08.080298 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.080196    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg" event={"ID":"d59de270-9763-49fa-8efc-625d58575e48","Type":"ContainerDied","Data":"35b3816932b30c2af46a88dc1ab8a87f7c6a5fd5d820114ca5daeb8377be4cdf"}
Oct 30 00:39:08.200499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.200227    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:39:08.296587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.295323    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-socket\") pod \"d59de270-9763-49fa-8efc-625d58575e48\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") "
Oct 30 00:39:08.296587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.295421    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-kube-api-access-vrkbz\" (UniqueName: \"kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-kube-api-access-vrkbz\") pod \"d59de270-9763-49fa-8efc-625d58575e48\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") "
Oct 30 00:39:08.296587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.295488    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-istio-token\" (UniqueName: \"kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-istio-token\") pod \"d59de270-9763-49fa-8efc-625d58575e48\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") "
Oct 30 00:39:08.296587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.295588    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59de270-9763-49fa-8efc-625d58575e48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59de270-9763-49fa-8efc-625d58575e48-istiod-ca-cert\") pod \"d59de270-9763-49fa-8efc-625d58575e48\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") "
Oct 30 00:39:08.296587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.295620    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-data\") pod \"d59de270-9763-49fa-8efc-625d58575e48\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") "
Oct 30 00:39:08.296587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.295657    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-certs\") pod \"d59de270-9763-49fa-8efc-625d58575e48\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") "
Oct 30 00:39:08.296587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.295722    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-envoy\") pod \"d59de270-9763-49fa-8efc-625d58575e48\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") "
Oct 30 00:39:08.296587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.295751    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d59de270-9763-49fa-8efc-625d58575e48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59de270-9763-49fa-8efc-625d58575e48-istio-podinfo\") pod \"d59de270-9763-49fa-8efc-625d58575e48\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") "
Oct 30 00:39:08.296587 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.295812    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-credential-socket\") pod \"d59de270-9763-49fa-8efc-625d58575e48\" (UID: \"d59de270-9763-49fa-8efc-625d58575e48\") "
Oct 30 00:39:08.297536 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.297298    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-credential-socket" pod "d59de270-9763-49fa-8efc-625d58575e48" (UID: "d59de270-9763-49fa-8efc-625d58575e48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:08.298006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.297685    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-socket" pod "d59de270-9763-49fa-8efc-625d58575e48" (UID: "d59de270-9763-49fa-8efc-625d58575e48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:08.298407 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.298385    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-data" pod "d59de270-9763-49fa-8efc-625d58575e48" (UID: "d59de270-9763-49fa-8efc-625d58575e48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:08.301701 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.301522    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-kube-api-access-vrkbz" pod "d59de270-9763-49fa-8efc-625d58575e48" (UID: "d59de270-9763-49fa-8efc-625d58575e48"). InnerVolumeSpecName "kube-api-access-vrkbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:08.302130 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.301849    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-certs" pod "d59de270-9763-49fa-8efc-625d58575e48" (UID: "d59de270-9763-49fa-8efc-625d58575e48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:08.308751 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.308552    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-istio-token" pod "d59de270-9763-49fa-8efc-625d58575e48" (UID: "d59de270-9763-49fa-8efc-625d58575e48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:08.309293 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.309001    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59de270-9763-49fa-8efc-625d58575e48-istiod-ca-cert" pod "d59de270-9763-49fa-8efc-625d58575e48" (UID: "d59de270-9763-49fa-8efc-625d58575e48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:39:08.310914 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.310892    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-envoy" pod "d59de270-9763-49fa-8efc-625d58575e48" (UID: "d59de270-9763-49fa-8efc-625d58575e48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:08.342797 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.342738    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d59de270-9763-49fa-8efc-625d58575e48-istio-podinfo" pod "d59de270-9763-49fa-8efc-625d58575e48" (UID: "d59de270-9763-49fa-8efc-625d58575e48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:39:08.397887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.397797    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59de270-9763-49fa-8efc-625d58575e48-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:08.397887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.397826    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:08.397887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.397838    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:08.397887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.397845    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:08.397887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.397854    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59de270-9763-49fa-8efc-625d58575e48-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:08.397887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.397864    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:08.397887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.397877    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59de270-9763-49fa-8efc-625d58575e48-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:08.397887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.397887    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrkbz\" (UniqueName: \"kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-kube-api-access-vrkbz\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:08.397887 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:08.397897    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d59de270-9763-49fa-8efc-625d58575e48-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:09.097645 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:09.097611    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"
Oct 30 00:39:09.098167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:09.097613    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg" event={"ID":"d59de270-9763-49fa-8efc-625d58575e48","Type":"ContainerDied","Data":"0230b2b5e1067ca3f8e2b3edea5149ce1034a99a267d70e6eafd1cae322ff891"}
Oct 30 00:39:09.098167 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:09.097762    4441 scope.go:122] "RemoveContainer" containerID="35b3816932b30c2af46a88dc1ab8a87f7c6a5fd5d820114ca5daeb8377be4cdf"
Oct 30 00:39:09.161295 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:09.161212    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"]
Oct 30 00:39:09.173368 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:09.172763    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-fb4fg"]
Oct 30 00:39:10.608933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:10.608900    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59de270-9763-49fa-8efc-625d58575e48" path="/var/lib/kubelet/pods/d59de270-9763-49fa-8efc-625d58575e48/volumes"
Oct 30 00:39:11.041265 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:11.041232    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"]
Oct 30 00:39:16.041614 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:16.041573    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm" podUID="13e3cf7a-4a8c-476e-93ec-2a1734dad886" containerName="payload-processing" containerID="cri-o://cae99119b8542e3d7d3ae8527046f883dd7506151b70a7fffe22e862a48e306f" gracePeriod=25
Oct 30 00:39:16.754166 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:16.754108    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main" containerID="cri-o://fcc1b88adea96d357c953f6c32483c4dbf58956079bd1c16f6813c3683952aa0" gracePeriod=45
Oct 30 00:39:16.760550 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:16.760514    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main" containerID="cri-o://22f753964d639f8c736b1a25a298ad7c14b98cc9fd771d6808272e7fa0643875" gracePeriod=45
Oct 30 00:39:17.029009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.028976    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:39:17.089261 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.089235    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:39:17.132503 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.132470    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tmp-dir\") pod \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") "
Oct 30 00:39:17.132769 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.132749    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7846274a-6b3c-41c5-ab26-45d2d75bc34f-kube-api-access-cfz9m\" (UniqueName: \"kubernetes.io/projected/7846274a-6b3c-41c5-ab26-45d2d75bc34f-kube-api-access-cfz9m\") pod \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") "
Oct 30 00:39:17.132894 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.132880    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-model-cache\") pod \"3ce95d81-779e-4c13-ab4b-04141c618347\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") "
Oct 30 00:39:17.133011 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.132998    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-tmp-dir\") pod \"3ce95d81-779e-4c13-ab4b-04141c618347\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") "
Oct 30 00:39:17.133176 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.133161    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce95d81-779e-4c13-ab4b-04141c618347-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce95d81-779e-4c13-ab4b-04141c618347-tls-certs\") pod \"3ce95d81-779e-4c13-ab4b-04141c618347\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") "
Oct 30 00:39:17.134224 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.134198    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce95d81-779e-4c13-ab4b-04141c618347-kube-api-access-8mjct\" (UniqueName: \"kubernetes.io/projected/3ce95d81-779e-4c13-ab4b-04141c618347-kube-api-access-8mjct\") pod \"3ce95d81-779e-4c13-ab4b-04141c618347\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") "
Oct 30 00:39:17.134418 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.134403    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-home\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-home\") pod \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") "
Oct 30 00:39:17.134549 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.134537    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-dshm\") pod \"3ce95d81-779e-4c13-ab4b-04141c618347\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") "
Oct 30 00:39:17.134677 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.134664    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-dshm\") pod \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") "
Oct 30 00:39:17.134869 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.134856    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-home\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-home\") pod \"3ce95d81-779e-4c13-ab4b-04141c618347\" (UID: \"3ce95d81-779e-4c13-ab4b-04141c618347\") "
Oct 30 00:39:17.134981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.134968    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-model-cache\") pod \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") "
Oct 30 00:39:17.135189 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.135174    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tls-certs\" (UniqueName: \"kubernetes.io/secret/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tls-certs\") pod \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\" (UID: \"7846274a-6b3c-41c5-ab26-45d2d75bc34f\") "
Oct 30 00:39:17.138059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.136921    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-model-cache" pod "3ce95d81-779e-4c13-ab4b-04141c618347" (UID: "3ce95d81-779e-4c13-ab4b-04141c618347"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:17.138059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.137131    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-model-cache" pod "7846274a-6b3c-41c5-ab26-45d2d75bc34f" (UID: "7846274a-6b3c-41c5-ab26-45d2d75bc34f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:17.138059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.137427    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-tmp-dir" pod "3ce95d81-779e-4c13-ab4b-04141c618347" (UID: "3ce95d81-779e-4c13-ab4b-04141c618347"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:17.138059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.137729    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-home" pod "3ce95d81-779e-4c13-ab4b-04141c618347" (UID: "3ce95d81-779e-4c13-ab4b-04141c618347"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:17.138059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.138000    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tmp-dir" pod "7846274a-6b3c-41c5-ab26-45d2d75bc34f" (UID: "7846274a-6b3c-41c5-ab26-45d2d75bc34f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:17.138868 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.138845    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-home" pod "7846274a-6b3c-41c5-ab26-45d2d75bc34f" (UID: "7846274a-6b3c-41c5-ab26-45d2d75bc34f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:17.142215 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.142152    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7846274a-6b3c-41c5-ab26-45d2d75bc34f-kube-api-access-cfz9m" pod "7846274a-6b3c-41c5-ab26-45d2d75bc34f" (UID: "7846274a-6b3c-41c5-ab26-45d2d75bc34f"). InnerVolumeSpecName "kube-api-access-cfz9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:17.146397 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.146365    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tls-certs" pod "7846274a-6b3c-41c5-ab26-45d2d75bc34f" (UID: "7846274a-6b3c-41c5-ab26-45d2d75bc34f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:39:17.153360 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.153329    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-dshm" pod "3ce95d81-779e-4c13-ab4b-04141c618347" (UID: "3ce95d81-779e-4c13-ab4b-04141c618347"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:17.153591 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.153565    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-dshm" pod "7846274a-6b3c-41c5-ab26-45d2d75bc34f" (UID: "7846274a-6b3c-41c5-ab26-45d2d75bc34f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:17.155471 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.154869    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce95d81-779e-4c13-ab4b-04141c618347-tls-certs" pod "3ce95d81-779e-4c13-ab4b-04141c618347" (UID: "3ce95d81-779e-4c13-ab4b-04141c618347"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:39:17.155471 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.154960    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce95d81-779e-4c13-ab4b-04141c618347-kube-api-access-8mjct" pod "3ce95d81-779e-4c13-ab4b-04141c618347" (UID: "3ce95d81-779e-4c13-ab4b-04141c618347"). InnerVolumeSpecName "kube-api-access-8mjct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:17.161055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.156836    4441 generic.go:356] "Generic (PLEG): container finished" podID="3ce95d81-779e-4c13-ab4b-04141c618347" containerID="22f753964d639f8c736b1a25a298ad7c14b98cc9fd771d6808272e7fa0643875" exitCode=0
Oct 30 00:39:17.161055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.156992    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" event={"ID":"3ce95d81-779e-4c13-ab4b-04141c618347","Type":"ContainerDied","Data":"22f753964d639f8c736b1a25a298ad7c14b98cc9fd771d6808272e7fa0643875"}
Oct 30 00:39:17.161055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.157057    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b" event={"ID":"3ce95d81-779e-4c13-ab4b-04141c618347","Type":"ContainerDied","Data":"7e4cef068166935e2f9f4390f42a22eebc0a38b8f2cabf1c40819e1cd07e12fb"}
Oct 30 00:39:17.161055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.157066    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"
Oct 30 00:39:17.161055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.157083    4441 scope.go:122] "RemoveContainer" containerID="22f753964d639f8c736b1a25a298ad7c14b98cc9fd771d6808272e7fa0643875"
Oct 30 00:39:17.167280 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.163958    4441 generic.go:356] "Generic (PLEG): container finished" podID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerID="fcc1b88adea96d357c953f6c32483c4dbf58956079bd1c16f6813c3683952aa0" exitCode=0
Oct 30 00:39:17.167280 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.164189    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"
Oct 30 00:39:17.167448 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.167339    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" event={"ID":"7846274a-6b3c-41c5-ab26-45d2d75bc34f","Type":"ContainerDied","Data":"fcc1b88adea96d357c953f6c32483c4dbf58956079bd1c16f6813c3683952aa0"}
Oct 30 00:39:17.167448 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.167395    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks" event={"ID":"7846274a-6b3c-41c5-ab26-45d2d75bc34f","Type":"ContainerDied","Data":"510771b1ee17ae61ca94e3210022e311331df60f58fed89207defc3e75536f7e"}
Oct 30 00:39:17.177044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.177005    4441 scope.go:122] "RemoveContainer" containerID="22f753964d639f8c736b1a25a298ad7c14b98cc9fd771d6808272e7fa0643875"
Oct 30 00:39:17.177418 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:17.177397    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f753964d639f8c736b1a25a298ad7c14b98cc9fd771d6808272e7fa0643875\": container with ID starting with 22f753964d639f8c736b1a25a298ad7c14b98cc9fd771d6808272e7fa0643875 not found: ID does not exist" containerID="22f753964d639f8c736b1a25a298ad7c14b98cc9fd771d6808272e7fa0643875"
Oct 30 00:39:17.177516 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.177428    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f753964d639f8c736b1a25a298ad7c14b98cc9fd771d6808272e7fa0643875"} err="failed to get container status \"22f753964d639f8c736b1a25a298ad7c14b98cc9fd771d6808272e7fa0643875\": rpc error: code = NotFound desc = could not find container \"22f753964d639f8c736b1a25a298ad7c14b98cc9fd771d6808272e7fa0643875\": container with ID starting with 22f753964d639f8c736b1a25a298ad7c14b98cc9fd771d6808272e7fa0643875 not found: ID does not exist"
Oct 30 00:39:17.177516 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.177452    4441 scope.go:122] "RemoveContainer" containerID="fcc1b88adea96d357c953f6c32483c4dbf58956079bd1c16f6813c3683952aa0"
Oct 30 00:39:17.188242 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.188227    4441 scope.go:122] "RemoveContainer" containerID="fcc1b88adea96d357c953f6c32483c4dbf58956079bd1c16f6813c3683952aa0"
Oct 30 00:39:17.188550 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:17.188530    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc1b88adea96d357c953f6c32483c4dbf58956079bd1c16f6813c3683952aa0\": container with ID starting with fcc1b88adea96d357c953f6c32483c4dbf58956079bd1c16f6813c3683952aa0 not found: ID does not exist" containerID="fcc1b88adea96d357c953f6c32483c4dbf58956079bd1c16f6813c3683952aa0"
Oct 30 00:39:17.188625 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.188559    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc1b88adea96d357c953f6c32483c4dbf58956079bd1c16f6813c3683952aa0"} err="failed to get container status \"fcc1b88adea96d357c953f6c32483c4dbf58956079bd1c16f6813c3683952aa0\": rpc error: code = NotFound desc = could not find container \"fcc1b88adea96d357c953f6c32483c4dbf58956079bd1c16f6813c3683952aa0\": container with ID starting with fcc1b88adea96d357c953f6c32483c4dbf58956079bd1c16f6813c3683952aa0 not found: ID does not exist"
Oct 30 00:39:17.220198 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.220177    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"]
Oct 30 00:39:17.224177 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.224154    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/distinct-w0-7505bd-kserve-7d77698984-46vks"]
Oct 30 00:39:17.237834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.237806    4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-tmp-dir\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:17.237834 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.237835    4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce95d81-779e-4c13-ab4b-04141c618347-tls-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:17.238009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.237846    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mjct\" (UniqueName: \"kubernetes.io/projected/3ce95d81-779e-4c13-ab4b-04141c618347-kube-api-access-8mjct\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:17.238009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.237854    4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-home\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:17.238009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.237864    4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-dshm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:17.238009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.237872    4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-dshm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:17.238009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.237879    4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-home\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:17.238009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.237886    4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-model-cache\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:17.238009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.237894    4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tls-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:17.238009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.237912    4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7846274a-6b3c-41c5-ab26-45d2d75bc34f-tmp-dir\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:17.238009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.237921    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfz9m\" (UniqueName: \"kubernetes.io/projected/7846274a-6b3c-41c5-ab26-45d2d75bc34f-kube-api-access-cfz9m\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:17.238009 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.237928    4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce95d81-779e-4c13-ab4b-04141c618347-model-cache\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:17.257435 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.257380    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"]
Oct 30 00:39:17.261241 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:17.261217    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/premium-w0-7505bd-kserve-7858c69c8f-v6c5b"]
Oct 30 00:39:18.178385 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:18.178312    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm" podUID="13e3cf7a-4a8c-476e-93ec-2a1734dad886" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:39:18.610487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:18.610455    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" path="/var/lib/kubelet/pods/3ce95d81-779e-4c13-ab4b-04141c618347/volumes"
Oct 30 00:39:18.611035 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:18.611001    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" path="/var/lib/kubelet/pods/7846274a-6b3c-41c5-ab26-45d2d75bc34f/volumes"
Oct 30 00:39:20.356509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.356461    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" containerID="cri-o://84deb3e5877df903fb4f1a91f39e8ab80f53baf9258ec75b7a46aaef1090b5e1" gracePeriod=45
Oct 30 00:39:20.368388 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.368342    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main" containerID="cri-o://3611ca84f1d8ba700823592149bfb4134e4450f8baa8973da4450306c06b7398" gracePeriod=45
Oct 30 00:39:20.417184 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.417146    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": dial tcp 10.133.0.77:8000: connect: connection refused"
Oct 30 00:39:20.651699 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.651668    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:39:20.688565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.688539    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:39:20.781216 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.781180    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929cb9ab-721f-4d60-b304-50aa1bab3573-kube-api-access-nj6j7\" (UniqueName: \"kubernetes.io/projected/929cb9ab-721f-4d60-b304-50aa1bab3573-kube-api-access-nj6j7\") pod \"929cb9ab-721f-4d60-b304-50aa1bab3573\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") "
Oct 30 00:39:20.781416 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.781238    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-home\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-home\") pod \"58d43eef-6285-4674-919c-20246cf8c597\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") "
Oct 30 00:39:20.781416 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.781277    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-model-cache\") pod \"929cb9ab-721f-4d60-b304-50aa1bab3573\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") "
Oct 30 00:39:20.781585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.781558    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-home" pod "58d43eef-6285-4674-919c-20246cf8c597" (UID: "58d43eef-6285-4674-919c-20246cf8c597"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:20.781656 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.781618    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-model-cache" pod "929cb9ab-721f-4d60-b304-50aa1bab3573" (UID: "929cb9ab-721f-4d60-b304-50aa1bab3573"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:20.781795 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.781748    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-dshm\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-dshm\") pod \"929cb9ab-721f-4d60-b304-50aa1bab3573\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") "
Oct 30 00:39:20.782164 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.782143    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-tmp-dir\") pod \"929cb9ab-721f-4d60-b304-50aa1bab3573\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") "
Oct 30 00:39:20.782276 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.782193    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-model-cache\") pod \"58d43eef-6285-4674-919c-20246cf8c597\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") "
Oct 30 00:39:20.782276 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.782268    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-dshm\") pod \"58d43eef-6285-4674-919c-20246cf8c597\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") "
Oct 30 00:39:20.782398 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.782336    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-home\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-home\") pod \"929cb9ab-721f-4d60-b304-50aa1bab3573\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") "
Oct 30 00:39:20.782398 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.782373    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/929cb9ab-721f-4d60-b304-50aa1bab3573-tls-certs\" (UniqueName: \"kubernetes.io/secret/929cb9ab-721f-4d60-b304-50aa1bab3573-tls-certs\") pod \"929cb9ab-721f-4d60-b304-50aa1bab3573\" (UID: \"929cb9ab-721f-4d60-b304-50aa1bab3573\") "
Oct 30 00:39:20.782532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.782513    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58d43eef-6285-4674-919c-20246cf8c597-tls-certs\" (UniqueName: \"kubernetes.io/secret/58d43eef-6285-4674-919c-20246cf8c597-tls-certs\") pod \"58d43eef-6285-4674-919c-20246cf8c597\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") "
Oct 30 00:39:20.782592 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.782555    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-tmp-dir\") pod \"58d43eef-6285-4674-919c-20246cf8c597\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") "
Oct 30 00:39:20.782645 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.782596    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d43eef-6285-4674-919c-20246cf8c597-kube-api-access-f7dlp\" (UniqueName: \"kubernetes.io/projected/58d43eef-6285-4674-919c-20246cf8c597-kube-api-access-f7dlp\") pod \"58d43eef-6285-4674-919c-20246cf8c597\" (UID: \"58d43eef-6285-4674-919c-20246cf8c597\") "
Oct 30 00:39:20.783488 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.783253    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-model-cache" pod "58d43eef-6285-4674-919c-20246cf8c597" (UID: "58d43eef-6285-4674-919c-20246cf8c597"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:20.783682 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.783651    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-tmp-dir" pod "58d43eef-6285-4674-919c-20246cf8c597" (UID: "58d43eef-6285-4674-919c-20246cf8c597"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:20.783941 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.783916    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-home" pod "929cb9ab-721f-4d60-b304-50aa1bab3573" (UID: "929cb9ab-721f-4d60-b304-50aa1bab3573"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:20.784161 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.784127    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-tmp-dir" pod "929cb9ab-721f-4d60-b304-50aa1bab3573" (UID: "929cb9ab-721f-4d60-b304-50aa1bab3573"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:20.784551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.784522    4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-home\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:20.784665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.784556    4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-tmp-dir\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:20.784665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.784568    4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-home\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:20.784665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.784579    4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-model-cache\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:20.784665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.784592    4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-tmp-dir\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:20.784665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.784602    4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-model-cache\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:20.786436 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.786400    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929cb9ab-721f-4d60-b304-50aa1bab3573-kube-api-access-nj6j7" pod "929cb9ab-721f-4d60-b304-50aa1bab3573" (UID: "929cb9ab-721f-4d60-b304-50aa1bab3573"). InnerVolumeSpecName "kube-api-access-nj6j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:20.786673 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.786648    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d43eef-6285-4674-919c-20246cf8c597-tls-certs" pod "58d43eef-6285-4674-919c-20246cf8c597" (UID: "58d43eef-6285-4674-919c-20246cf8c597"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:39:20.794202 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.786977    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929cb9ab-721f-4d60-b304-50aa1bab3573-tls-certs" pod "929cb9ab-721f-4d60-b304-50aa1bab3573" (UID: "929cb9ab-721f-4d60-b304-50aa1bab3573"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:39:20.794202 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.787136    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-dshm" pod "58d43eef-6285-4674-919c-20246cf8c597" (UID: "58d43eef-6285-4674-919c-20246cf8c597"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:20.794202 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.789933    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d43eef-6285-4674-919c-20246cf8c597-kube-api-access-f7dlp" pod "58d43eef-6285-4674-919c-20246cf8c597" (UID: "58d43eef-6285-4674-919c-20246cf8c597"). InnerVolumeSpecName "kube-api-access-f7dlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:20.794202 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.790365    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-dshm" pod "929cb9ab-721f-4d60-b304-50aa1bab3573" (UID: "929cb9ab-721f-4d60-b304-50aa1bab3573"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:20.887652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.886605    4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929cb9ab-721f-4d60-b304-50aa1bab3573-dshm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:20.887652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.886646    4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d43eef-6285-4674-919c-20246cf8c597-dshm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:20.887652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.886663    4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929cb9ab-721f-4d60-b304-50aa1bab3573-tls-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:20.887652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.886677    4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d43eef-6285-4674-919c-20246cf8c597-tls-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:20.887652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.886690    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7dlp\" (UniqueName: \"kubernetes.io/projected/58d43eef-6285-4674-919c-20246cf8c597-kube-api-access-f7dlp\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:20.887652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:20.886706    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj6j7\" (UniqueName: \"kubernetes.io/projected/929cb9ab-721f-4d60-b304-50aa1bab3573-kube-api-access-nj6j7\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:21.203900 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.203824    4441 generic.go:356] "Generic (PLEG): container finished" podID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerID="3611ca84f1d8ba700823592149bfb4134e4450f8baa8973da4450306c06b7398" exitCode=0
Oct 30 00:39:21.204112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.203897    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"
Oct 30 00:39:21.204112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.203903    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" event={"ID":"929cb9ab-721f-4d60-b304-50aa1bab3573","Type":"ContainerDied","Data":"3611ca84f1d8ba700823592149bfb4134e4450f8baa8973da4450306c06b7398"}
Oct 30 00:39:21.204112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.203953    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw" event={"ID":"929cb9ab-721f-4d60-b304-50aa1bab3573","Type":"ContainerDied","Data":"29a9c34a01f952945d22556cbb1d9ad0638f246daf42c3c8c9d354a19060a624"}
Oct 30 00:39:21.204112 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.203977    4441 scope.go:122] "RemoveContainer" containerID="3611ca84f1d8ba700823592149bfb4134e4450f8baa8973da4450306c06b7398"
Oct 30 00:39:21.205786 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.205756    4441 generic.go:356] "Generic (PLEG): container finished" podID="58d43eef-6285-4674-919c-20246cf8c597" containerID="84deb3e5877df903fb4f1a91f39e8ab80f53baf9258ec75b7a46aaef1090b5e1" exitCode=0
Oct 30 00:39:21.205904 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.205803    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"
Oct 30 00:39:21.205904 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.205793    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" event={"ID":"58d43eef-6285-4674-919c-20246cf8c597","Type":"ContainerDied","Data":"84deb3e5877df903fb4f1a91f39e8ab80f53baf9258ec75b7a46aaef1090b5e1"}
Oct 30 00:39:21.206061 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.205919    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg" event={"ID":"58d43eef-6285-4674-919c-20246cf8c597","Type":"ContainerDied","Data":"a12cce445b23f97db85688c265dc7248a6ce32de145dd4494e356d94e75c0a75"}
Oct 30 00:39:21.215207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.215185    4441 scope.go:122] "RemoveContainer" containerID="3611ca84f1d8ba700823592149bfb4134e4450f8baa8973da4450306c06b7398"
Oct 30 00:39:21.215660 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:21.215639    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3611ca84f1d8ba700823592149bfb4134e4450f8baa8973da4450306c06b7398\": container with ID starting with 3611ca84f1d8ba700823592149bfb4134e4450f8baa8973da4450306c06b7398 not found: ID does not exist" containerID="3611ca84f1d8ba700823592149bfb4134e4450f8baa8973da4450306c06b7398"
Oct 30 00:39:21.215746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.215678    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3611ca84f1d8ba700823592149bfb4134e4450f8baa8973da4450306c06b7398"} err="failed to get container status \"3611ca84f1d8ba700823592149bfb4134e4450f8baa8973da4450306c06b7398\": rpc error: code = NotFound desc = could not find container \"3611ca84f1d8ba700823592149bfb4134e4450f8baa8973da4450306c06b7398\": container with ID starting with 3611ca84f1d8ba700823592149bfb4134e4450f8baa8973da4450306c06b7398 not found: ID does not exist"
Oct 30 00:39:21.215746 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.215694    4441 scope.go:122] "RemoveContainer" containerID="84deb3e5877df903fb4f1a91f39e8ab80f53baf9258ec75b7a46aaef1090b5e1"
Oct 30 00:39:21.225530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.225513    4441 scope.go:122] "RemoveContainer" containerID="84deb3e5877df903fb4f1a91f39e8ab80f53baf9258ec75b7a46aaef1090b5e1"
Oct 30 00:39:21.225804 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:21.225784    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84deb3e5877df903fb4f1a91f39e8ab80f53baf9258ec75b7a46aaef1090b5e1\": container with ID starting with 84deb3e5877df903fb4f1a91f39e8ab80f53baf9258ec75b7a46aaef1090b5e1 not found: ID does not exist" containerID="84deb3e5877df903fb4f1a91f39e8ab80f53baf9258ec75b7a46aaef1090b5e1"
Oct 30 00:39:21.225877 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.225814    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84deb3e5877df903fb4f1a91f39e8ab80f53baf9258ec75b7a46aaef1090b5e1"} err="failed to get container status \"84deb3e5877df903fb4f1a91f39e8ab80f53baf9258ec75b7a46aaef1090b5e1\": rpc error: code = NotFound desc = could not find container \"84deb3e5877df903fb4f1a91f39e8ab80f53baf9258ec75b7a46aaef1090b5e1\": container with ID starting with 84deb3e5877df903fb4f1a91f39e8ab80f53baf9258ec75b7a46aaef1090b5e1 not found: ID does not exist"
Oct 30 00:39:21.252836 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.252812    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"]
Oct 30 00:39:21.256687 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.256667    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/premium-w2-322d20-kserve-86dfb66bc9-5qrlw"]
Oct 30 00:39:21.287859 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.287833    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"]
Oct 30 00:39:21.291801 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:21.291779    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/distinct-2-w2-322d20-kserve-657769d5b5-fd7hg"]
Oct 30 00:39:22.609619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:22.609575    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d43eef-6285-4674-919c-20246cf8c597" path="/var/lib/kubelet/pods/58d43eef-6285-4674-919c-20246cf8c597/volumes"
Oct 30 00:39:22.610296 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:22.610277    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" path="/var/lib/kubelet/pods/929cb9ab-721f-4d60-b304-50aa1bab3573/volumes"
Oct 30 00:39:24.609248 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:24.609220    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:39:28.178830 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:28.178793    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm" podUID="13e3cf7a-4a8c-476e-93ec-2a1734dad886" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:39:30.902656 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:30.902613    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"]
Oct 30 00:39:30.903229 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:30.902865    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk" podUID="fe79b3ad-1c24-4dde-bc30-23a4264e35f9" containerName="istio-proxy" containerID="cri-o://d3ebaebd640f60d2f2c002670b7ec7d7c1b21f1e2d9daa07951a29173106bab5" gracePeriod=30
Oct 30 00:39:30.913744 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:30.913718    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"]
Oct 30 00:39:30.913961 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:30.913938    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd" podUID="e14e3e15-a207-4782-9905-befa94ac0ddc" containerName="istio-proxy" containerID="cri-o://9a3fa24bdd8e3e163a0f7d91611f2dd5f812bf75e1a8b7b9fbfa8c79128ed5e9" gracePeriod=30
Oct 30 00:39:33.901907 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:33.901868    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:39:36.295046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.290378    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:39:36.321353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.321286    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:39:36.322608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.322579    4441 generic.go:356] "Generic (PLEG): container finished" podID="13e3cf7a-4a8c-476e-93ec-2a1734dad886" containerID="cae99119b8542e3d7d3ae8527046f883dd7506151b70a7fffe22e862a48e306f" exitCode=0
Oct 30 00:39:36.322792 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.322712    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm" event={"ID":"13e3cf7a-4a8c-476e-93ec-2a1734dad886","Type":"ContainerDied","Data":"cae99119b8542e3d7d3ae8527046f883dd7506151b70a7fffe22e862a48e306f"}
Oct 30 00:39:36.324221 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.324199    4441 generic.go:356] "Generic (PLEG): container finished" podID="fe79b3ad-1c24-4dde-bc30-23a4264e35f9" containerID="d3ebaebd640f60d2f2c002670b7ec7d7c1b21f1e2d9daa07951a29173106bab5" exitCode=0
Oct 30 00:39:36.324449 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.324250    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"
Oct 30 00:39:36.324531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.324265    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk" event={"ID":"fe79b3ad-1c24-4dde-bc30-23a4264e35f9","Type":"ContainerDied","Data":"d3ebaebd640f60d2f2c002670b7ec7d7c1b21f1e2d9daa07951a29173106bab5"}
Oct 30 00:39:36.324598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.324531    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk" event={"ID":"fe79b3ad-1c24-4dde-bc30-23a4264e35f9","Type":"ContainerDied","Data":"c60e696764b8428326b679d4297dd4826d3cb41f248ad3032cd058a1e08e1e37"}
Oct 30 00:39:36.324598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.324564    4441 scope.go:122] "RemoveContainer" containerID="d3ebaebd640f60d2f2c002670b7ec7d7c1b21f1e2d9daa07951a29173106bab5"
Oct 30 00:39:36.326819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.326563    4441 generic.go:356] "Generic (PLEG): container finished" podID="e14e3e15-a207-4782-9905-befa94ac0ddc" containerID="9a3fa24bdd8e3e163a0f7d91611f2dd5f812bf75e1a8b7b9fbfa8c79128ed5e9" exitCode=0
Oct 30 00:39:36.326819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.326604    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd" event={"ID":"e14e3e15-a207-4782-9905-befa94ac0ddc","Type":"ContainerDied","Data":"9a3fa24bdd8e3e163a0f7d91611f2dd5f812bf75e1a8b7b9fbfa8c79128ed5e9"}
Oct 30 00:39:36.326819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.326631    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd" event={"ID":"e14e3e15-a207-4782-9905-befa94ac0ddc","Type":"ContainerDied","Data":"fbaadad22e3b1f1fc21617e4b1faa130932c5de4abeca442128abe4e22d7041e"}
Oct 30 00:39:36.326819 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.326700    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"
Oct 30 00:39:36.335703 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.335636    4441 scope.go:122] "RemoveContainer" containerID="d3ebaebd640f60d2f2c002670b7ec7d7c1b21f1e2d9daa07951a29173106bab5"
Oct 30 00:39:36.336168 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:36.336013    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3ebaebd640f60d2f2c002670b7ec7d7c1b21f1e2d9daa07951a29173106bab5\": container with ID starting with d3ebaebd640f60d2f2c002670b7ec7d7c1b21f1e2d9daa07951a29173106bab5 not found: ID does not exist" containerID="d3ebaebd640f60d2f2c002670b7ec7d7c1b21f1e2d9daa07951a29173106bab5"
Oct 30 00:39:36.336168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.336067    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ebaebd640f60d2f2c002670b7ec7d7c1b21f1e2d9daa07951a29173106bab5"} err="failed to get container status \"d3ebaebd640f60d2f2c002670b7ec7d7c1b21f1e2d9daa07951a29173106bab5\": rpc error: code = NotFound desc = could not find container \"d3ebaebd640f60d2f2c002670b7ec7d7c1b21f1e2d9daa07951a29173106bab5\": container with ID starting with d3ebaebd640f60d2f2c002670b7ec7d7c1b21f1e2d9daa07951a29173106bab5 not found: ID does not exist"
Oct 30 00:39:36.336168 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.336089    4441 scope.go:122] "RemoveContainer" containerID="9a3fa24bdd8e3e163a0f7d91611f2dd5f812bf75e1a8b7b9fbfa8c79128ed5e9"
Oct 30 00:39:36.346471 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.346452    4441 scope.go:122] "RemoveContainer" containerID="9a3fa24bdd8e3e163a0f7d91611f2dd5f812bf75e1a8b7b9fbfa8c79128ed5e9"
Oct 30 00:39:36.346731 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:36.346713    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3fa24bdd8e3e163a0f7d91611f2dd5f812bf75e1a8b7b9fbfa8c79128ed5e9\": container with ID starting with 9a3fa24bdd8e3e163a0f7d91611f2dd5f812bf75e1a8b7b9fbfa8c79128ed5e9 not found: ID does not exist" containerID="9a3fa24bdd8e3e163a0f7d91611f2dd5f812bf75e1a8b7b9fbfa8c79128ed5e9"
Oct 30 00:39:36.346807 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.346748    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3fa24bdd8e3e163a0f7d91611f2dd5f812bf75e1a8b7b9fbfa8c79128ed5e9"} err="failed to get container status \"9a3fa24bdd8e3e163a0f7d91611f2dd5f812bf75e1a8b7b9fbfa8c79128ed5e9\": rpc error: code = NotFound desc = could not find container \"9a3fa24bdd8e3e163a0f7d91611f2dd5f812bf75e1a8b7b9fbfa8c79128ed5e9\": container with ID starting with 9a3fa24bdd8e3e163a0f7d91611f2dd5f812bf75e1a8b7b9fbfa8c79128ed5e9 not found: ID does not exist"
Oct 30 00:39:36.409003 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.408976    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"
Oct 30 00:39:36.414121 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414091    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e3cf7a-4a8c-476e-93ec-2a1734dad886-kube-api-access-g89zt\" (UniqueName: \"kubernetes.io/projected/13e3cf7a-4a8c-476e-93ec-2a1734dad886-kube-api-access-g89zt\") pod \"13e3cf7a-4a8c-476e-93ec-2a1734dad886\" (UID: \"13e3cf7a-4a8c-476e-93ec-2a1734dad886\") "
Oct 30 00:39:36.414258 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414137    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e3cf7a-4a8c-476e-93ec-2a1734dad886-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e3cf7a-4a8c-476e-93ec-2a1734dad886-plugins-config-volume\") pod \"13e3cf7a-4a8c-476e-93ec-2a1734dad886\" (UID: \"13e3cf7a-4a8c-476e-93ec-2a1734dad886\") "
Oct 30 00:39:36.414258 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414166    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-certs\") pod \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") "
Oct 30 00:39:36.414444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414420    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-certs" pod "fe79b3ad-1c24-4dde-bc30-23a4264e35f9" (UID: "fe79b3ad-1c24-4dde-bc30-23a4264e35f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:36.414505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414442    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-envoy\") pod \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") "
Oct 30 00:39:36.414505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414496    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-certs\") pod \"e14e3e15-a207-4782-9905-befa94ac0ddc\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") "
Oct 30 00:39:36.414611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414555    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-envoy\") pod \"e14e3e15-a207-4782-9905-befa94ac0ddc\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") "
Oct 30 00:39:36.414611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414590    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e14e3e15-a207-4782-9905-befa94ac0ddc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14e3e15-a207-4782-9905-befa94ac0ddc-istiod-ca-cert\") pod \"e14e3e15-a207-4782-9905-befa94ac0ddc\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") "
Oct 30 00:39:36.414611 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414597    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e3cf7a-4a8c-476e-93ec-2a1734dad886-plugins-config-volume" pod "13e3cf7a-4a8c-476e-93ec-2a1734dad886" (UID: "13e3cf7a-4a8c-476e-93ec-2a1734dad886"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:39:36.414774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414617    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-data\") pod \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") "
Oct 30 00:39:36.414774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414648    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istiod-ca-cert\") pod \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") "
Oct 30 00:39:36.414774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414698    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-kube-api-access-htwls\" (UniqueName: \"kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-kube-api-access-htwls\") pod \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") "
Oct 30 00:39:36.414774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414730    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-podinfo\") pod \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") "
Oct 30 00:39:36.414981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414769    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-podinfo\") pod \"e14e3e15-a207-4782-9905-befa94ac0ddc\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") "
Oct 30 00:39:36.414981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414852    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-socket\") pod \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") "
Oct 30 00:39:36.414981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414919    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-kube-api-access-q9fcd\" (UniqueName: \"kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-kube-api-access-q9fcd\") pod \"e14e3e15-a207-4782-9905-befa94ac0ddc\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") "
Oct 30 00:39:36.414981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.414953    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-credential-socket\") pod \"e14e3e15-a207-4782-9905-befa94ac0ddc\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") "
Oct 30 00:39:36.415233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.415015    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-token\" (UniqueName: \"kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-token\") pod \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") "
Oct 30 00:39:36.415233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.415113    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-data\") pod \"e14e3e15-a207-4782-9905-befa94ac0ddc\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") "
Oct 30 00:39:36.415233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.415173    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-credential-socket\") pod \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\" (UID: \"fe79b3ad-1c24-4dde-bc30-23a4264e35f9\") "
Oct 30 00:39:36.415233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.415218    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-token\" (UniqueName: \"kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-token\") pod \"e14e3e15-a207-4782-9905-befa94ac0ddc\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") "
Oct 30 00:39:36.415434 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.415272    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-socket\") pod \"e14e3e15-a207-4782-9905-befa94ac0ddc\" (UID: \"e14e3e15-a207-4782-9905-befa94ac0ddc\") "
Oct 30 00:39:36.415505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.415481    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-certs" pod "e14e3e15-a207-4782-9905-befa94ac0ddc" (UID: "e14e3e15-a207-4782-9905-befa94ac0ddc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:36.415697 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.415678    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-data" pod "fe79b3ad-1c24-4dde-bc30-23a4264e35f9" (UID: "fe79b3ad-1c24-4dde-bc30-23a4264e35f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:36.415785 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.415689    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-socket" pod "e14e3e15-a207-4782-9905-befa94ac0ddc" (UID: "e14e3e15-a207-4782-9905-befa94ac0ddc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:36.415921 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.415902    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-credential-socket" pod "e14e3e15-a207-4782-9905-befa94ac0ddc" (UID: "e14e3e15-a207-4782-9905-befa94ac0ddc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:36.416007 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.415985    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14e3e15-a207-4782-9905-befa94ac0ddc-istiod-ca-cert" pod "e14e3e15-a207-4782-9905-befa94ac0ddc" (UID: "e14e3e15-a207-4782-9905-befa94ac0ddc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:39:36.417257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.416467    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.417257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.416492    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.417257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.416507    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e3cf7a-4a8c-476e-93ec-2a1734dad886-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.417257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.416520    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.417257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.416538    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.417257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.416550    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14e3e15-a207-4782-9905-befa94ac0ddc-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.417257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.416563    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.417257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.416517    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e3cf7a-4a8c-476e-93ec-2a1734dad886-kube-api-access-g89zt" pod "13e3cf7a-4a8c-476e-93ec-2a1734dad886" (UID: "13e3cf7a-4a8c-476e-93ec-2a1734dad886"). InnerVolumeSpecName "kube-api-access-g89zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:36.417257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.416735    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-data" pod "e14e3e15-a207-4782-9905-befa94ac0ddc" (UID: "e14e3e15-a207-4782-9905-befa94ac0ddc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:36.417257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.416856    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-credential-socket" pod "fe79b3ad-1c24-4dde-bc30-23a4264e35f9" (UID: "fe79b3ad-1c24-4dde-bc30-23a4264e35f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:36.417257 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.416910    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-envoy" pod "fe79b3ad-1c24-4dde-bc30-23a4264e35f9" (UID: "fe79b3ad-1c24-4dde-bc30-23a4264e35f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:36.418450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.418252    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istiod-ca-cert" pod "fe79b3ad-1c24-4dde-bc30-23a4264e35f9" (UID: "fe79b3ad-1c24-4dde-bc30-23a4264e35f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:39:36.418450 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.418409    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-token" pod "fe79b3ad-1c24-4dde-bc30-23a4264e35f9" (UID: "fe79b3ad-1c24-4dde-bc30-23a4264e35f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:36.418661 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.418633    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-socket" pod "fe79b3ad-1c24-4dde-bc30-23a4264e35f9" (UID: "fe79b3ad-1c24-4dde-bc30-23a4264e35f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:36.418896 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.418875    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-kube-api-access-q9fcd" pod "e14e3e15-a207-4782-9905-befa94ac0ddc" (UID: "e14e3e15-a207-4782-9905-befa94ac0ddc"). InnerVolumeSpecName "kube-api-access-q9fcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:36.420415 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.420356    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-token" pod "e14e3e15-a207-4782-9905-befa94ac0ddc" (UID: "e14e3e15-a207-4782-9905-befa94ac0ddc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:36.420896 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.420876    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-envoy" pod "e14e3e15-a207-4782-9905-befa94ac0ddc" (UID: "e14e3e15-a207-4782-9905-befa94ac0ddc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:36.421128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.421103    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-podinfo" pod "e14e3e15-a207-4782-9905-befa94ac0ddc" (UID: "e14e3e15-a207-4782-9905-befa94ac0ddc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:39:36.421128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.421118    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-kube-api-access-htwls" pod "fe79b3ad-1c24-4dde-bc30-23a4264e35f9" (UID: "fe79b3ad-1c24-4dde-bc30-23a4264e35f9"). InnerVolumeSpecName "kube-api-access-htwls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:36.421498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.421479    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-podinfo" pod "fe79b3ad-1c24-4dde-bc30-23a4264e35f9" (UID: "fe79b3ad-1c24-4dde-bc30-23a4264e35f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:39:36.517940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.517895    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.517940 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.517935    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9fcd\" (UniqueName: \"kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-kube-api-access-q9fcd\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.518240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.517952    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.518240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.517966    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.518240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.517978    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.518240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.517991    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.518240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.518005    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g89zt\" (UniqueName: \"kubernetes.io/projected/13e3cf7a-4a8c-476e-93ec-2a1734dad886-kube-api-access-g89zt\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.518240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.518017    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.518240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.518045    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.518240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.518059    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.518240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.518071    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htwls\" (UniqueName: \"kubernetes.io/projected/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-kube-api-access-htwls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.518240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.518084    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe79b3ad-1c24-4dde-bc30-23a4264e35f9-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.518240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.518097    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14e3e15-a207-4782-9905-befa94ac0ddc-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:36.672131 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.672069    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"]
Oct 30 00:39:36.677273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.677246    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-cgktd"]
Oct 30 00:39:36.709471 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.709442    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"]
Oct 30 00:39:36.713019 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:36.712995    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmbmk"]
Oct 30 00:39:37.335107 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:37.335068    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm" event={"ID":"13e3cf7a-4a8c-476e-93ec-2a1734dad886","Type":"ContainerDied","Data":"1b15e83f28ec4c8a95c8b1e448cc2a265277177257131cb1b19cb7fa1f7e9de5"}
Oct 30 00:39:37.335623 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:37.335111    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"
Oct 30 00:39:37.335623 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:37.335160    4441 scope.go:122] "RemoveContainer" containerID="cae99119b8542e3d7d3ae8527046f883dd7506151b70a7fffe22e862a48e306f"
Oct 30 00:39:37.396645 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:37.396614    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"]
Oct 30 00:39:37.400105 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:37.399455    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"]
Oct 30 00:39:37.400105 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:37.399698    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w" podUID="6606ae96-4a55-400c-8108-fd38d17c5529" containerName="istio-proxy" containerID="cri-o://88edabdb9186bab7355c2df984c70c0a20d856c172cabecf4a7d8ac6425558f3" gracePeriod=30
Oct 30 00:39:37.404177 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:37.403854    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-322d20-7b74788b48-9dwrm"]
Oct 30 00:39:37.406840 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:37.406819    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"]
Oct 30 00:39:37.407046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:37.407003    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q" podUID="d4141111-103c-47f4-ac95-1b0d19fbd5cd" containerName="istio-proxy" containerID="cri-o://b43ff405c64139920fc694facc5d0d099e89026dd1f98a1f8abc29b45dbd5520" gracePeriod=30
Oct 30 00:39:37.416992 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:37.410199    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"]
Oct 30 00:39:37.416992 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:37.410491    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8" podUID="a3ba27e0-c510-41a6-97ec-83d0c5b637ec" containerName="istio-proxy" containerID="cri-o://22b3d74c6ddee56cd5a91906871a8e13ea445db114cb04c2a21c454984bf84da" gracePeriod=30
Oct 30 00:39:38.157644 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.157601    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"]
Oct 30 00:39:38.157828 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.157804    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs" podUID="0e2143da-e83f-4fad-a3fe-7509571c484e" containerName="maas-api" containerID="cri-o://8b0a7c572b9d5994735b003b7aae41399a2b74bb74c374715a8644d2d3a4eea8" gracePeriod=30
Oct 30 00:39:38.351660 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.351624    4441 generic.go:356] "Generic (PLEG): container finished" podID="0e2143da-e83f-4fad-a3fe-7509571c484e" containerID="8b0a7c572b9d5994735b003b7aae41399a2b74bb74c374715a8644d2d3a4eea8" exitCode=0
Oct 30 00:39:38.352222 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.351760    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs" event={"ID":"0e2143da-e83f-4fad-a3fe-7509571c484e","Type":"ContainerDied","Data":"8b0a7c572b9d5994735b003b7aae41399a2b74bb74c374715a8644d2d3a4eea8"}
Oct 30 00:39:38.479955 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.479923    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:39:38.545667 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.545637    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-maas-api-tls\") pod \"0e2143da-e83f-4fad-a3fe-7509571c484e\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") "
Oct 30 00:39:38.545947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.545750    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e2143da-e83f-4fad-a3fe-7509571c484e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2143da-e83f-4fad-a3fe-7509571c484e-openshift-service-ca\") pod \"0e2143da-e83f-4fad-a3fe-7509571c484e\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") "
Oct 30 00:39:38.545947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.545786    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e2143da-e83f-4fad-a3fe-7509571c484e-kube-api-access-22gvn\" (UniqueName: \"kubernetes.io/projected/0e2143da-e83f-4fad-a3fe-7509571c484e-kube-api-access-22gvn\") pod \"0e2143da-e83f-4fad-a3fe-7509571c484e\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") "
Oct 30 00:39:38.545947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.545838    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-metrics-certs\") pod \"0e2143da-e83f-4fad-a3fe-7509571c484e\" (UID: \"0e2143da-e83f-4fad-a3fe-7509571c484e\") "
Oct 30 00:39:38.546410 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.546386    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2143da-e83f-4fad-a3fe-7509571c484e-openshift-service-ca" pod "0e2143da-e83f-4fad-a3fe-7509571c484e" (UID: "0e2143da-e83f-4fad-a3fe-7509571c484e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:39:38.548263 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.548222    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-metrics-certs" pod "0e2143da-e83f-4fad-a3fe-7509571c484e" (UID: "0e2143da-e83f-4fad-a3fe-7509571c484e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:39:38.548503 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.548477    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2143da-e83f-4fad-a3fe-7509571c484e-kube-api-access-22gvn" pod "0e2143da-e83f-4fad-a3fe-7509571c484e" (UID: "0e2143da-e83f-4fad-a3fe-7509571c484e"). InnerVolumeSpecName "kube-api-access-22gvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:38.548799 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.548760    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-maas-api-tls" pod "0e2143da-e83f-4fad-a3fe-7509571c484e" (UID: "0e2143da-e83f-4fad-a3fe-7509571c484e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:39:38.610128 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.610083    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e3cf7a-4a8c-476e-93ec-2a1734dad886" path="/var/lib/kubelet/pods/13e3cf7a-4a8c-476e-93ec-2a1734dad886/volumes"
Oct 30 00:39:38.610668 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.610645    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14e3e15-a207-4782-9905-befa94ac0ddc" path="/var/lib/kubelet/pods/e14e3e15-a207-4782-9905-befa94ac0ddc/volumes"
Oct 30 00:39:38.611189 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.611170    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe79b3ad-1c24-4dde-bc30-23a4264e35f9" path="/var/lib/kubelet/pods/fe79b3ad-1c24-4dde-bc30-23a4264e35f9/volumes"
Oct 30 00:39:38.670040 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.669931    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:38.670040 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.669966    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2143da-e83f-4fad-a3fe-7509571c484e-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:38.670040 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.669982    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22gvn\" (UniqueName: \"kubernetes.io/projected/0e2143da-e83f-4fad-a3fe-7509571c484e-kube-api-access-22gvn\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:38.670040 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:38.669996    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2143da-e83f-4fad-a3fe-7509571c484e-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:39.359272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:39.359239    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"
Oct 30 00:39:39.359272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:39.359259    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs" event={"ID":"0e2143da-e83f-4fad-a3fe-7509571c484e","Type":"ContainerDied","Data":"2c670981ea1748b0da8c22ff8a62809ad2aba3f183f48a3e5a765bcdc9804156"}
Oct 30 00:39:39.359774 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:39.359327    4441 scope.go:122] "RemoveContainer" containerID="8b0a7c572b9d5994735b003b7aae41399a2b74bb74c374715a8644d2d3a4eea8"
Oct 30 00:39:39.409265 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:39.409241    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"]
Oct 30 00:39:39.420409 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:39.420389    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d96837-5f8b5886fd-9dfbs"]
Oct 30 00:39:40.608143 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:40.608107    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e2143da-e83f-4fad-a3fe-7509571c484e" path="/var/lib/kubelet/pods/0e2143da-e83f-4fad-a3fe-7509571c484e/volumes"
Oct 30 00:39:42.743182 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.741775    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:39:42.758758 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.758730    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.819156    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826131    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-certs\") pod \"6606ae96-4a55-400c-8108-fd38d17c5529\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826173    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-data\") pod \"6606ae96-4a55-400c-8108-fd38d17c5529\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826217    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-envoy\") pod \"6606ae96-4a55-400c-8108-fd38d17c5529\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826289    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-socket\") pod \"6606ae96-4a55-400c-8108-fd38d17c5529\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826318    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6606ae96-4a55-400c-8108-fd38d17c5529-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6606ae96-4a55-400c-8108-fd38d17c5529-istiod-ca-cert\") pod \"6606ae96-4a55-400c-8108-fd38d17c5529\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826354    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-kube-api-access-bxvwp\" (UniqueName: \"kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-kube-api-access-bxvwp\") pod \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826384    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-certs" pod "6606ae96-4a55-400c-8108-fd38d17c5529" (UID: "6606ae96-4a55-400c-8108-fd38d17c5529"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826424    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-credential-socket\") pod \"6606ae96-4a55-400c-8108-fd38d17c5529\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826456    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-socket\") pod \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826497    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-istio-token\" (UniqueName: \"kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-istio-token\") pod \"6606ae96-4a55-400c-8108-fd38d17c5529\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826535    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-kube-api-access-k6qdv\" (UniqueName: \"kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-kube-api-access-k6qdv\") pod \"6606ae96-4a55-400c-8108-fd38d17c5529\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826621    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-certs\") pod \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826643    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-socket" pod "6606ae96-4a55-400c-8108-fd38d17c5529" (UID: "6606ae96-4a55-400c-8108-fd38d17c5529"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826647    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-podinfo\") pod \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826722    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istiod-ca-cert\") pod \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826830    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-credential-socket\") pod \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826962    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6606ae96-4a55-400c-8108-fd38d17c5529-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6606ae96-4a55-400c-8108-fd38d17c5529-istio-podinfo\") pod \"6606ae96-4a55-400c-8108-fd38d17c5529\" (UID: \"6606ae96-4a55-400c-8108-fd38d17c5529\") "
Oct 30 00:39:42.827047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.826992    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-data\") pod \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") "
Oct 30 00:39:42.828231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.827070    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-envoy\") pod \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") "
Oct 30 00:39:42.828231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.827128    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-token\" (UniqueName: \"kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-token\") pod \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\" (UID: \"d4141111-103c-47f4-ac95-1b0d19fbd5cd\") "
Oct 30 00:39:42.828231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.827306    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-credential-socket" pod "d4141111-103c-47f4-ac95-1b0d19fbd5cd" (UID: "d4141111-103c-47f4-ac95-1b0d19fbd5cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.828231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.827617    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6606ae96-4a55-400c-8108-fd38d17c5529-istiod-ca-cert" pod "6606ae96-4a55-400c-8108-fd38d17c5529" (UID: "6606ae96-4a55-400c-8108-fd38d17c5529"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:39:42.828231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.827715    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istiod-ca-cert" pod "d4141111-103c-47f4-ac95-1b0d19fbd5cd" (UID: "d4141111-103c-47f4-ac95-1b0d19fbd5cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:39:42.828231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.827884    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.828231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.827960    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.828231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.827977    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.828231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.827992    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.828231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.828021    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6606ae96-4a55-400c-8108-fd38d17c5529-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.828231 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.828058    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-data" pod "6606ae96-4a55-400c-8108-fd38d17c5529" (UID: "6606ae96-4a55-400c-8108-fd38d17c5529"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.862311 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.828380    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-data" pod "d4141111-103c-47f4-ac95-1b0d19fbd5cd" (UID: "d4141111-103c-47f4-ac95-1b0d19fbd5cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.862311 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.834390    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-credential-socket" pod "6606ae96-4a55-400c-8108-fd38d17c5529" (UID: "6606ae96-4a55-400c-8108-fd38d17c5529"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.862311 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.834661    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-socket" pod "d4141111-103c-47f4-ac95-1b0d19fbd5cd" (UID: "d4141111-103c-47f4-ac95-1b0d19fbd5cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.862311 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.835695    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-certs" pod "d4141111-103c-47f4-ac95-1b0d19fbd5cd" (UID: "d4141111-103c-47f4-ac95-1b0d19fbd5cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.862311 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.845577    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-podinfo" pod "d4141111-103c-47f4-ac95-1b0d19fbd5cd" (UID: "d4141111-103c-47f4-ac95-1b0d19fbd5cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:39:42.878084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.878046    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-kube-api-access-k6qdv" pod "6606ae96-4a55-400c-8108-fd38d17c5529" (UID: "6606ae96-4a55-400c-8108-fd38d17c5529"). InnerVolumeSpecName "kube-api-access-k6qdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:42.878503 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.878440    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-envoy" pod "d4141111-103c-47f4-ac95-1b0d19fbd5cd" (UID: "d4141111-103c-47f4-ac95-1b0d19fbd5cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.878503 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.878443    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6606ae96-4a55-400c-8108-fd38d17c5529-istio-podinfo" pod "6606ae96-4a55-400c-8108-fd38d17c5529" (UID: "6606ae96-4a55-400c-8108-fd38d17c5529"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:39:42.878708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.878546    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-kube-api-access-bxvwp" pod "d4141111-103c-47f4-ac95-1b0d19fbd5cd" (UID: "d4141111-103c-47f4-ac95-1b0d19fbd5cd"). InnerVolumeSpecName "kube-api-access-bxvwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:42.878708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.878643    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-istio-token" pod "6606ae96-4a55-400c-8108-fd38d17c5529" (UID: "6606ae96-4a55-400c-8108-fd38d17c5529"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:42.879199 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.879165    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-envoy" pod "6606ae96-4a55-400c-8108-fd38d17c5529" (UID: "6606ae96-4a55-400c-8108-fd38d17c5529"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.880356 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.880251    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-token" pod "d4141111-103c-47f4-ac95-1b0d19fbd5cd" (UID: "d4141111-103c-47f4-ac95-1b0d19fbd5cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:42.928732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.928693    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-certs\") pod \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") "
Oct 30 00:39:42.928904 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.928789    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-envoy\") pod \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") "
Oct 30 00:39:42.928904 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.928827    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-socket\") pod \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") "
Oct 30 00:39:42.928904 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.928885    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istiod-ca-cert\") pod \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") "
Oct 30 00:39:42.929065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.928934    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-credential-socket\") pod \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") "
Oct 30 00:39:42.929065 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929049    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-data\") pod \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") "
Oct 30 00:39:42.929192 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929095    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-podinfo\") pod \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") "
Oct 30 00:39:42.929192 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929125    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-kube-api-access-4dq4t\" (UniqueName: \"kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-kube-api-access-4dq4t\") pod \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") "
Oct 30 00:39:42.929192 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929163    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-token\" (UniqueName: \"kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-token\") pod \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\" (UID: \"a3ba27e0-c510-41a6-97ec-83d0c5b637ec\") "
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929404    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-data" pod "a3ba27e0-c510-41a6-97ec-83d0c5b637ec" (UID: "a3ba27e0-c510-41a6-97ec-83d0c5b637ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929456    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-certs" pod "a3ba27e0-c510-41a6-97ec-83d0c5b637ec" (UID: "a3ba27e0-c510-41a6-97ec-83d0c5b637ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929520    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-credential-socket" pod "a3ba27e0-c510-41a6-97ec-83d0c5b637ec" (UID: "a3ba27e0-c510-41a6-97ec-83d0c5b637ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929571    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929759    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-socket" pod "a3ba27e0-c510-41a6-97ec-83d0c5b637ec" (UID: "a3ba27e0-c510-41a6-97ec-83d0c5b637ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929781    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929799    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929813    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6qdv\" (UniqueName: \"kubernetes.io/projected/6606ae96-4a55-400c-8108-fd38d17c5529-kube-api-access-k6qdv\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929827    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929841    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929854    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6606ae96-4a55-400c-8108-fd38d17c5529-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929867    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929880    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929896    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929909    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929921    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6606ae96-4a55-400c-8108-fd38d17c5529-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.930566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.929934    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxvwp\" (UniqueName: \"kubernetes.io/projected/d4141111-103c-47f4-ac95-1b0d19fbd5cd-kube-api-access-bxvwp\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:42.931604 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.930628    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istiod-ca-cert" pod "a3ba27e0-c510-41a6-97ec-83d0c5b637ec" (UID: "a3ba27e0-c510-41a6-97ec-83d0c5b637ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:39:42.931944 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.931919    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-token" pod "a3ba27e0-c510-41a6-97ec-83d0c5b637ec" (UID: "a3ba27e0-c510-41a6-97ec-83d0c5b637ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:42.933240 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.933210    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-envoy" pod "a3ba27e0-c510-41a6-97ec-83d0c5b637ec" (UID: "a3ba27e0-c510-41a6-97ec-83d0c5b637ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:42.933378 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.933262    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-podinfo" pod "a3ba27e0-c510-41a6-97ec-83d0c5b637ec" (UID: "a3ba27e0-c510-41a6-97ec-83d0c5b637ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:39:42.933498 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:42.933476    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-kube-api-access-4dq4t" pod "a3ba27e0-c510-41a6-97ec-83d0c5b637ec" (UID: "a3ba27e0-c510-41a6-97ec-83d0c5b637ec"). InnerVolumeSpecName "kube-api-access-4dq4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:43.031330 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.031297    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:43.031330 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.031325    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:43.031330 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.031335    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:43.031562 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.031344    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:43.031562 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.031353    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:43.031562 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.031361    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dq4t\" (UniqueName: \"kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-kube-api-access-4dq4t\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:43.031562 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.031371    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:43.031562 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.031379    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:43.031562 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.031387    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ba27e0-c510-41a6-97ec-83d0c5b637ec-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:43.418132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.418036    4441 generic.go:356] "Generic (PLEG): container finished" podID="d4141111-103c-47f4-ac95-1b0d19fbd5cd" containerID="b43ff405c64139920fc694facc5d0d099e89026dd1f98a1f8abc29b45dbd5520" exitCode=0
Oct 30 00:39:43.418132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.418095    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"
Oct 30 00:39:43.418132 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.418100    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q" event={"ID":"d4141111-103c-47f4-ac95-1b0d19fbd5cd","Type":"ContainerDied","Data":"b43ff405c64139920fc694facc5d0d099e89026dd1f98a1f8abc29b45dbd5520"}
Oct 30 00:39:43.418432 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.418187    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q" event={"ID":"d4141111-103c-47f4-ac95-1b0d19fbd5cd","Type":"ContainerDied","Data":"542c729268f881d59cb861204db9c27eeffa8db98696cfa8fd60e6a1d8cd55ce"}
Oct 30 00:39:43.418432 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.418218    4441 scope.go:122] "RemoveContainer" containerID="b43ff405c64139920fc694facc5d0d099e89026dd1f98a1f8abc29b45dbd5520"
Oct 30 00:39:43.420853 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.420827    4441 generic.go:356] "Generic (PLEG): container finished" podID="a3ba27e0-c510-41a6-97ec-83d0c5b637ec" containerID="22b3d74c6ddee56cd5a91906871a8e13ea445db114cb04c2a21c454984bf84da" exitCode=0
Oct 30 00:39:43.421005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.420897    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"
Oct 30 00:39:43.421005 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.420963    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8" event={"ID":"a3ba27e0-c510-41a6-97ec-83d0c5b637ec","Type":"ContainerDied","Data":"22b3d74c6ddee56cd5a91906871a8e13ea445db114cb04c2a21c454984bf84da"}
Oct 30 00:39:43.421184 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.421108    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8" event={"ID":"a3ba27e0-c510-41a6-97ec-83d0c5b637ec","Type":"ContainerDied","Data":"f312c143b2d73c550bb31a90b6cb5dc7fa3cbf13703ffd1c35d098aa8e3c3e34"}
Oct 30 00:39:43.425060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.422951    4441 generic.go:356] "Generic (PLEG): container finished" podID="6606ae96-4a55-400c-8108-fd38d17c5529" containerID="88edabdb9186bab7355c2df984c70c0a20d856c172cabecf4a7d8ac6425558f3" exitCode=0
Oct 30 00:39:43.425060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.422979    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w" event={"ID":"6606ae96-4a55-400c-8108-fd38d17c5529","Type":"ContainerDied","Data":"88edabdb9186bab7355c2df984c70c0a20d856c172cabecf4a7d8ac6425558f3"}
Oct 30 00:39:43.425060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.423054    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w" event={"ID":"6606ae96-4a55-400c-8108-fd38d17c5529","Type":"ContainerDied","Data":"892d92236fe22c1f90a110b46338923ce536d6dcdce7e7d3ce64a17309f288c1"}
Oct 30 00:39:43.425060 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.423063    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"
Oct 30 00:39:43.451433 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.451409    4441 scope.go:122] "RemoveContainer" containerID="b43ff405c64139920fc694facc5d0d099e89026dd1f98a1f8abc29b45dbd5520"
Oct 30 00:39:43.451748 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:43.451726    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b43ff405c64139920fc694facc5d0d099e89026dd1f98a1f8abc29b45dbd5520\": container with ID starting with b43ff405c64139920fc694facc5d0d099e89026dd1f98a1f8abc29b45dbd5520 not found: ID does not exist" containerID="b43ff405c64139920fc694facc5d0d099e89026dd1f98a1f8abc29b45dbd5520"
Oct 30 00:39:43.451827 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.451756    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b43ff405c64139920fc694facc5d0d099e89026dd1f98a1f8abc29b45dbd5520"} err="failed to get container status \"b43ff405c64139920fc694facc5d0d099e89026dd1f98a1f8abc29b45dbd5520\": rpc error: code = NotFound desc = could not find container \"b43ff405c64139920fc694facc5d0d099e89026dd1f98a1f8abc29b45dbd5520\": container with ID starting with b43ff405c64139920fc694facc5d0d099e89026dd1f98a1f8abc29b45dbd5520 not found: ID does not exist"
Oct 30 00:39:43.451827 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.451772    4441 scope.go:122] "RemoveContainer" containerID="22b3d74c6ddee56cd5a91906871a8e13ea445db114cb04c2a21c454984bf84da"
Oct 30 00:39:43.461329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.461307    4441 scope.go:122] "RemoveContainer" containerID="22b3d74c6ddee56cd5a91906871a8e13ea445db114cb04c2a21c454984bf84da"
Oct 30 00:39:43.461611 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:43.461590    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b3d74c6ddee56cd5a91906871a8e13ea445db114cb04c2a21c454984bf84da\": container with ID starting with 22b3d74c6ddee56cd5a91906871a8e13ea445db114cb04c2a21c454984bf84da not found: ID does not exist" containerID="22b3d74c6ddee56cd5a91906871a8e13ea445db114cb04c2a21c454984bf84da"
Oct 30 00:39:43.461665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.461619    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b3d74c6ddee56cd5a91906871a8e13ea445db114cb04c2a21c454984bf84da"} err="failed to get container status \"22b3d74c6ddee56cd5a91906871a8e13ea445db114cb04c2a21c454984bf84da\": rpc error: code = NotFound desc = could not find container \"22b3d74c6ddee56cd5a91906871a8e13ea445db114cb04c2a21c454984bf84da\": container with ID starting with 22b3d74c6ddee56cd5a91906871a8e13ea445db114cb04c2a21c454984bf84da not found: ID does not exist"
Oct 30 00:39:43.461665 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.461636    4441 scope.go:122] "RemoveContainer" containerID="88edabdb9186bab7355c2df984c70c0a20d856c172cabecf4a7d8ac6425558f3"
Oct 30 00:39:43.473179 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.473159    4441 scope.go:122] "RemoveContainer" containerID="88edabdb9186bab7355c2df984c70c0a20d856c172cabecf4a7d8ac6425558f3"
Oct 30 00:39:43.473456 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:43.473435    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88edabdb9186bab7355c2df984c70c0a20d856c172cabecf4a7d8ac6425558f3\": container with ID starting with 88edabdb9186bab7355c2df984c70c0a20d856c172cabecf4a7d8ac6425558f3 not found: ID does not exist" containerID="88edabdb9186bab7355c2df984c70c0a20d856c172cabecf4a7d8ac6425558f3"
Oct 30 00:39:43.473532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.473467    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88edabdb9186bab7355c2df984c70c0a20d856c172cabecf4a7d8ac6425558f3"} err="failed to get container status \"88edabdb9186bab7355c2df984c70c0a20d856c172cabecf4a7d8ac6425558f3\": rpc error: code = NotFound desc = could not find container \"88edabdb9186bab7355c2df984c70c0a20d856c172cabecf4a7d8ac6425558f3\": container with ID starting with 88edabdb9186bab7355c2df984c70c0a20d856c172cabecf4a7d8ac6425558f3 not found: ID does not exist"
Oct 30 00:39:43.500084 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.500017    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"]
Oct 30 00:39:43.510959 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.510936    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xrb8q"]
Oct 30 00:39:43.537054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.537019    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"]
Oct 30 00:39:43.542835 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.542810    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vvpf8"]
Oct 30 00:39:43.575480 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.575459    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"]
Oct 30 00:39:43.579674 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:43.579646    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-94t7w"]
Oct 30 00:39:44.607219 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:44.607180    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6606ae96-4a55-400c-8108-fd38d17c5529" path="/var/lib/kubelet/pods/6606ae96-4a55-400c-8108-fd38d17c5529/volumes"
Oct 30 00:39:44.607837 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:44.607815    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ba27e0-c510-41a6-97ec-83d0c5b637ec" path="/var/lib/kubelet/pods/a3ba27e0-c510-41a6-97ec-83d0c5b637ec/volumes"
Oct 30 00:39:44.608326 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:44.608306    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4141111-103c-47f4-ac95-1b0d19fbd5cd" path="/var/lib/kubelet/pods/d4141111-103c-47f4-ac95-1b0d19fbd5cd/volumes"
Oct 30 00:39:45.088530 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:45.088492    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:39:54.192534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:54.192500    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"]
Oct 30 00:39:54.192927 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:54.192716    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z" podUID="e27789fe-70fb-4912-8873-aabb7694db35" containerName="istio-proxy" containerID="cri-o://af2c25ae150a2438ebb9227bc9c97fbe24f8f2354cef6adad834310caeeb1102" gracePeriod=30
Oct 30 00:39:56.949648 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:56.949621    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:39:56.959162 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:56.959140    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sf6zz_d74677bd-f7f3-4a55-b5e7-0216819f5400/ovn-acl-logging/0.log"
Oct 30 00:39:57.620423 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:57.620379    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"]
Oct 30 00:39:57.620618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:57.620594    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz" podUID="1d5c887e-c6d9-41b0-8b98-78428e123af5" containerName="maas-api" containerID="cri-o://73adc4398cc61c562ed4925e87f5cd9bd60c2ce8c48abc822d90a75f6f05ff86" gracePeriod=30
Oct 30 00:39:57.763965 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:57.763923    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"]
Oct 30 00:39:57.782142 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:57.782107    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"]
Oct 30 00:39:57.914246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:57.914219    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:39:58.093047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.089182    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-maas-api-tls\") pod \"1d5c887e-c6d9-41b0-8b98-78428e123af5\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") "
Oct 30 00:39:58.093047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.089239    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d5c887e-c6d9-41b0-8b98-78428e123af5-kube-api-access-vwstf\" (UniqueName: \"kubernetes.io/projected/1d5c887e-c6d9-41b0-8b98-78428e123af5-kube-api-access-vwstf\") pod \"1d5c887e-c6d9-41b0-8b98-78428e123af5\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") "
Oct 30 00:39:58.093047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.090573    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d5c887e-c6d9-41b0-8b98-78428e123af5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5c887e-c6d9-41b0-8b98-78428e123af5-openshift-service-ca\") pod \"1d5c887e-c6d9-41b0-8b98-78428e123af5\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") "
Oct 30 00:39:58.093047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.091199    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5c887e-c6d9-41b0-8b98-78428e123af5-openshift-service-ca" pod "1d5c887e-c6d9-41b0-8b98-78428e123af5" (UID: "1d5c887e-c6d9-41b0-8b98-78428e123af5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:39:58.093047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.091282    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-metrics-certs\") pod \"1d5c887e-c6d9-41b0-8b98-78428e123af5\" (UID: \"1d5c887e-c6d9-41b0-8b98-78428e123af5\") "
Oct 30 00:39:58.093047 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.092728    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5c887e-c6d9-41b0-8b98-78428e123af5-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:58.103154 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.103043    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-maas-api-tls" pod "1d5c887e-c6d9-41b0-8b98-78428e123af5" (UID: "1d5c887e-c6d9-41b0-8b98-78428e123af5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:39:58.103402 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.103370    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-metrics-certs" pod "1d5c887e-c6d9-41b0-8b98-78428e123af5" (UID: "1d5c887e-c6d9-41b0-8b98-78428e123af5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:39:58.116312 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.116275    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5c887e-c6d9-41b0-8b98-78428e123af5-kube-api-access-vwstf" pod "1d5c887e-c6d9-41b0-8b98-78428e123af5" (UID: "1d5c887e-c6d9-41b0-8b98-78428e123af5"). InnerVolumeSpecName "kube-api-access-vwstf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:58.193394 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.193318    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:58.193394 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.193348    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwstf\" (UniqueName: \"kubernetes.io/projected/1d5c887e-c6d9-41b0-8b98-78428e123af5-kube-api-access-vwstf\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:58.193394 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.193361    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5c887e-c6d9-41b0-8b98-78428e123af5-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:58.511471 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.511436    4441 generic.go:356] "Generic (PLEG): container finished" podID="1d5c887e-c6d9-41b0-8b98-78428e123af5" containerID="73adc4398cc61c562ed4925e87f5cd9bd60c2ce8c48abc822d90a75f6f05ff86" exitCode=0
Oct 30 00:39:58.511637 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.511470    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz" event={"ID":"1d5c887e-c6d9-41b0-8b98-78428e123af5","Type":"ContainerDied","Data":"73adc4398cc61c562ed4925e87f5cd9bd60c2ce8c48abc822d90a75f6f05ff86"}
Oct 30 00:39:58.511637 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.511487    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"
Oct 30 00:39:58.511637 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.511527    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz" event={"ID":"1d5c887e-c6d9-41b0-8b98-78428e123af5","Type":"ContainerDied","Data":"da10f4363e0ad05bdcf8192b29d261b8993a9e0abb2852467d24a817db099f78"}
Oct 30 00:39:58.511637 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.511551    4441 scope.go:122] "RemoveContainer" containerID="73adc4398cc61c562ed4925e87f5cd9bd60c2ce8c48abc822d90a75f6f05ff86"
Oct 30 00:39:58.520933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.520916    4441 scope.go:122] "RemoveContainer" containerID="73adc4398cc61c562ed4925e87f5cd9bd60c2ce8c48abc822d90a75f6f05ff86"
Oct 30 00:39:58.521206 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:58.521189    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73adc4398cc61c562ed4925e87f5cd9bd60c2ce8c48abc822d90a75f6f05ff86\": container with ID starting with 73adc4398cc61c562ed4925e87f5cd9bd60c2ce8c48abc822d90a75f6f05ff86 not found: ID does not exist" containerID="73adc4398cc61c562ed4925e87f5cd9bd60c2ce8c48abc822d90a75f6f05ff86"
Oct 30 00:39:58.521267 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.521212    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73adc4398cc61c562ed4925e87f5cd9bd60c2ce8c48abc822d90a75f6f05ff86"} err="failed to get container status \"73adc4398cc61c562ed4925e87f5cd9bd60c2ce8c48abc822d90a75f6f05ff86\": rpc error: code = NotFound desc = could not find container \"73adc4398cc61c562ed4925e87f5cd9bd60c2ce8c48abc822d90a75f6f05ff86\": container with ID starting with 73adc4398cc61c562ed4925e87f5cd9bd60c2ce8c48abc822d90a75f6f05ff86 not found: ID does not exist"
Oct 30 00:39:58.560204 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.560180    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"]
Oct 30 00:39:58.574169 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.574143    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-63e288-5d687f9c4f-fn8jz"]
Oct 30 00:39:58.608509 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:58.608483    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d5c887e-c6d9-41b0-8b98-78428e123af5" path="/var/lib/kubelet/pods/1d5c887e-c6d9-41b0-8b98-78428e123af5/volumes"
Oct 30 00:39:59.474470 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.474404    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:39:59.506298 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.506265    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e27789fe-70fb-4912-8873-aabb7694db35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27789fe-70fb-4912-8873-aabb7694db35-istio-podinfo\") pod \"e27789fe-70fb-4912-8873-aabb7694db35\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") "
Oct 30 00:39:59.506442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.506314    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-envoy\") pod \"e27789fe-70fb-4912-8873-aabb7694db35\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") "
Oct 30 00:39:59.506442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.506351    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-certs\") pod \"e27789fe-70fb-4912-8873-aabb7694db35\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") "
Oct 30 00:39:59.506442 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.506423    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-data\") pod \"e27789fe-70fb-4912-8873-aabb7694db35\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") "
Oct 30 00:39:59.506613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.506487    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-kube-api-access-8vjcc\" (UniqueName: \"kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-kube-api-access-8vjcc\") pod \"e27789fe-70fb-4912-8873-aabb7694db35\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") "
Oct 30 00:39:59.506613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.506516    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-socket\") pod \"e27789fe-70fb-4912-8873-aabb7694db35\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") "
Oct 30 00:39:59.506613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.506542    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e27789fe-70fb-4912-8873-aabb7694db35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27789fe-70fb-4912-8873-aabb7694db35-istiod-ca-cert\") pod \"e27789fe-70fb-4912-8873-aabb7694db35\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") "
Oct 30 00:39:59.506613 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.506589    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-certs" pod "e27789fe-70fb-4912-8873-aabb7694db35" (UID: "e27789fe-70fb-4912-8873-aabb7694db35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:59.506959 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.506623    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-istio-token\" (UniqueName: \"kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-istio-token\") pod \"e27789fe-70fb-4912-8873-aabb7694db35\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") "
Oct 30 00:39:59.506959 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.506685    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-data" pod "e27789fe-70fb-4912-8873-aabb7694db35" (UID: "e27789fe-70fb-4912-8873-aabb7694db35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:59.506959 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.506772    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-socket" pod "e27789fe-70fb-4912-8873-aabb7694db35" (UID: "e27789fe-70fb-4912-8873-aabb7694db35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:59.506959 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.506948    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27789fe-70fb-4912-8873-aabb7694db35-istiod-ca-cert" pod "e27789fe-70fb-4912-8873-aabb7694db35" (UID: "e27789fe-70fb-4912-8873-aabb7694db35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:39:59.507207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.507094    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:59.507207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.507107    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:59.507207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.507115    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:59.507207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.507123    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27789fe-70fb-4912-8873-aabb7694db35-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:59.508567 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.508537    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e27789fe-70fb-4912-8873-aabb7694db35-istio-podinfo" pod "e27789fe-70fb-4912-8873-aabb7694db35" (UID: "e27789fe-70fb-4912-8873-aabb7694db35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:39:59.508567 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.508556    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-kube-api-access-8vjcc" pod "e27789fe-70fb-4912-8873-aabb7694db35" (UID: "e27789fe-70fb-4912-8873-aabb7694db35"). InnerVolumeSpecName "kube-api-access-8vjcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:59.508750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.508609    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-envoy" pod "e27789fe-70fb-4912-8873-aabb7694db35" (UID: "e27789fe-70fb-4912-8873-aabb7694db35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:59.508750 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.508723    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-istio-token" pod "e27789fe-70fb-4912-8873-aabb7694db35" (UID: "e27789fe-70fb-4912-8873-aabb7694db35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:59.518801 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.518775    4441 generic.go:356] "Generic (PLEG): container finished" podID="e27789fe-70fb-4912-8873-aabb7694db35" containerID="af2c25ae150a2438ebb9227bc9c97fbe24f8f2354cef6adad834310caeeb1102" exitCode=0
Oct 30 00:39:59.518889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.518819    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"
Oct 30 00:39:59.518889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.518850    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z" event={"ID":"e27789fe-70fb-4912-8873-aabb7694db35","Type":"ContainerDied","Data":"af2c25ae150a2438ebb9227bc9c97fbe24f8f2354cef6adad834310caeeb1102"}
Oct 30 00:39:59.518978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.518906    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z" event={"ID":"e27789fe-70fb-4912-8873-aabb7694db35","Type":"ContainerDied","Data":"2b6385531efca2ee0938acdb560c190c55ded62bbc892cb404ddf7ca7900a2fe"}
Oct 30 00:39:59.518978 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.518937    4441 scope.go:122] "RemoveContainer" containerID="af2c25ae150a2438ebb9227bc9c97fbe24f8f2354cef6adad834310caeeb1102"
Oct 30 00:39:59.528189 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.528168    4441 scope.go:122] "RemoveContainer" containerID="af2c25ae150a2438ebb9227bc9c97fbe24f8f2354cef6adad834310caeeb1102"
Oct 30 00:39:59.528489 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:39:59.528472    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af2c25ae150a2438ebb9227bc9c97fbe24f8f2354cef6adad834310caeeb1102\": container with ID starting with af2c25ae150a2438ebb9227bc9c97fbe24f8f2354cef6adad834310caeeb1102 not found: ID does not exist" containerID="af2c25ae150a2438ebb9227bc9c97fbe24f8f2354cef6adad834310caeeb1102"
Oct 30 00:39:59.528570 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.528496    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2c25ae150a2438ebb9227bc9c97fbe24f8f2354cef6adad834310caeeb1102"} err="failed to get container status \"af2c25ae150a2438ebb9227bc9c97fbe24f8f2354cef6adad834310caeeb1102\": rpc error: code = NotFound desc = could not find container \"af2c25ae150a2438ebb9227bc9c97fbe24f8f2354cef6adad834310caeeb1102\": container with ID starting with af2c25ae150a2438ebb9227bc9c97fbe24f8f2354cef6adad834310caeeb1102 not found: ID does not exist"
Oct 30 00:39:59.608206 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.608177    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-credential-socket\") pod \"e27789fe-70fb-4912-8873-aabb7694db35\" (UID: \"e27789fe-70fb-4912-8873-aabb7694db35\") "
Oct 30 00:39:59.608479 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.608457    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-credential-socket" pod "e27789fe-70fb-4912-8873-aabb7694db35" (UID: "e27789fe-70fb-4912-8873-aabb7694db35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:59.608597 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.608584    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:59.608638 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.608601    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:59.608638 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.608611    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27789fe-70fb-4912-8873-aabb7694db35-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:59.608638 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.608620    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27789fe-70fb-4912-8873-aabb7694db35-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:59.608638 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.608628    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjcc\" (UniqueName: \"kubernetes.io/projected/e27789fe-70fb-4912-8873-aabb7694db35-kube-api-access-8vjcc\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:39:59.864340 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.864305    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"]
Oct 30 00:39:59.869631 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:39:59.869601    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wrv7z"]
Oct 30 00:40:00.608002 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:00.607969    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27789fe-70fb-4912-8873-aabb7694db35" path="/var/lib/kubelet/pods/e27789fe-70fb-4912-8873-aabb7694db35/volumes"
Oct 30 00:40:02.601049 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:02.600622    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:40:02.765360 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:02.765310    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872" podUID="187932ee-ad15-4170-9e9f-13dc02fa8756" containerName="payload-processing" containerID="cri-o://c8e1b4083554026cd0d7626531434c00e80ecaa08a0ac3a220409e960cf30a59" gracePeriod=25
Oct 30 00:40:02.782875 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:02.782820    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg" podUID="6df19eba-a75b-49a8-9c52-fd7000d43e75" containerName="payload-pre-processing" containerID="cri-o://d7899ddfce98043e050200e85ec196631fd0110d46fbbb21e4becb4fcc08b1ee" gracePeriod=25
Oct 30 00:40:03.900451 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:40:03.900414    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:40:04.375500 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:04.375464    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg" podUID="6df19eba-a75b-49a8-9c52-fd7000d43e75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:40:04.375666 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:04.375467    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872" podUID="187932ee-ad15-4170-9e9f-13dc02fa8756" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:40:14.375971 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:14.375931    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872" podUID="187932ee-ad15-4170-9e9f-13dc02fa8756" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:40:14.376572 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:14.375938    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg" podUID="6df19eba-a75b-49a8-9c52-fd7000d43e75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:40:17.096523 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:17.096487    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"]
Oct 30 00:40:18.036720 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.036682    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"]
Oct 30 00:40:18.036905 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.036874    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5" podUID="b439da36-a99a-4847-9884-1c19d9d0b8e8" containerName="maas-api" containerID="cri-o://6e58a3f112470779ee285b4d1f3857674b91f5a74285967f6e8156720f5e3c47" gracePeriod=30
Oct 30 00:40:18.309160 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.309132    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:40:18.397933 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.397905    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-maas-api-tls\") pod \"b439da36-a99a-4847-9884-1c19d9d0b8e8\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") "
Oct 30 00:40:18.398134 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.398034    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-metrics-certs\") pod \"b439da36-a99a-4847-9884-1c19d9d0b8e8\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") "
Oct 30 00:40:18.398134 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.398078    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b439da36-a99a-4847-9884-1c19d9d0b8e8-kube-api-access-gcxcz\" (UniqueName: \"kubernetes.io/projected/b439da36-a99a-4847-9884-1c19d9d0b8e8-kube-api-access-gcxcz\") pod \"b439da36-a99a-4847-9884-1c19d9d0b8e8\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") "
Oct 30 00:40:18.398269 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.398212    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b439da36-a99a-4847-9884-1c19d9d0b8e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b439da36-a99a-4847-9884-1c19d9d0b8e8-openshift-service-ca\") pod \"b439da36-a99a-4847-9884-1c19d9d0b8e8\" (UID: \"b439da36-a99a-4847-9884-1c19d9d0b8e8\") "
Oct 30 00:40:18.398585 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.398554    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b439da36-a99a-4847-9884-1c19d9d0b8e8-openshift-service-ca" pod "b439da36-a99a-4847-9884-1c19d9d0b8e8" (UID: "b439da36-a99a-4847-9884-1c19d9d0b8e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:18.400250 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.400223    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-maas-api-tls" pod "b439da36-a99a-4847-9884-1c19d9d0b8e8" (UID: "b439da36-a99a-4847-9884-1c19d9d0b8e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:18.400372 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.400340    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b439da36-a99a-4847-9884-1c19d9d0b8e8-kube-api-access-gcxcz" pod "b439da36-a99a-4847-9884-1c19d9d0b8e8" (UID: "b439da36-a99a-4847-9884-1c19d9d0b8e8"). InnerVolumeSpecName "kube-api-access-gcxcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:40:18.400472 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.400451    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-metrics-certs" pod "b439da36-a99a-4847-9884-1c19d9d0b8e8" (UID: "b439da36-a99a-4847-9884-1c19d9d0b8e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:18.500057 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.500009    4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-metrics-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:18.500246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.500058    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcxcz\" (UniqueName: \"kubernetes.io/projected/b439da36-a99a-4847-9884-1c19d9d0b8e8-kube-api-access-gcxcz\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:18.500246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.500085    4441 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b439da36-a99a-4847-9884-1c19d9d0b8e8-openshift-service-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:18.500246 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.500094    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b439da36-a99a-4847-9884-1c19d9d0b8e8-maas-api-tls\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:18.619706 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.619680    4441 generic.go:356] "Generic (PLEG): container finished" podID="b439da36-a99a-4847-9884-1c19d9d0b8e8" containerID="6e58a3f112470779ee285b4d1f3857674b91f5a74285967f6e8156720f5e3c47" exitCode=0
Oct 30 00:40:18.619841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.619726    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"
Oct 30 00:40:18.619841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.619761    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5" event={"ID":"b439da36-a99a-4847-9884-1c19d9d0b8e8","Type":"ContainerDied","Data":"6e58a3f112470779ee285b4d1f3857674b91f5a74285967f6e8156720f5e3c47"}
Oct 30 00:40:18.619841 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.619819    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5" event={"ID":"b439da36-a99a-4847-9884-1c19d9d0b8e8","Type":"ContainerDied","Data":"bf774bea881ed1e7be17253233dccb81b37d36ca3bd7768522dfb86877696bf1"}
Oct 30 00:40:18.619960 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.619848    4441 scope.go:122] "RemoveContainer" containerID="6e58a3f112470779ee285b4d1f3857674b91f5a74285967f6e8156720f5e3c47"
Oct 30 00:40:18.629314 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.629295    4441 scope.go:122] "RemoveContainer" containerID="6e58a3f112470779ee285b4d1f3857674b91f5a74285967f6e8156720f5e3c47"
Oct 30 00:40:18.629572 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:40:18.629552    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e58a3f112470779ee285b4d1f3857674b91f5a74285967f6e8156720f5e3c47\": container with ID starting with 6e58a3f112470779ee285b4d1f3857674b91f5a74285967f6e8156720f5e3c47 not found: ID does not exist" containerID="6e58a3f112470779ee285b4d1f3857674b91f5a74285967f6e8156720f5e3c47"
Oct 30 00:40:18.629628 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.629579    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e58a3f112470779ee285b4d1f3857674b91f5a74285967f6e8156720f5e3c47"} err="failed to get container status \"6e58a3f112470779ee285b4d1f3857674b91f5a74285967f6e8156720f5e3c47\": rpc error: code = NotFound desc = could not find container \"6e58a3f112470779ee285b4d1f3857674b91f5a74285967f6e8156720f5e3c47\": container with ID starting with 6e58a3f112470779ee285b4d1f3857674b91f5a74285967f6e8156720f5e3c47 not found: ID does not exist"
Oct 30 00:40:18.668889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.668841    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"]
Oct 30 00:40:18.671602 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:18.671579    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f1268c-7b547884f6-bkbl5"]
Oct 30 00:40:20.607755 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:20.607722    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b439da36-a99a-4847-9884-1c19d9d0b8e8" path="/var/lib/kubelet/pods/b439da36-a99a-4847-9884-1c19d9d0b8e8/volumes"
Oct 30 00:40:21.573256 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:21.573221    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:40:22.097267 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:22.097228    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j" podUID="2af359bf-d670-4f66-9dab-2a95b328705e" containerName="payload-processing" containerID="cri-o://1b4abd5ac1e811073f76877a80755345a7c5080e78060a1ec1c95891d48a2e02" gracePeriod=25
Oct 30 00:40:22.350724 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:22.350638    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j" podUID="2af359bf-d670-4f66-9dab-2a95b328705e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:40:23.044191 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.044124    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"
Oct 30 00:40:23.074255 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.074234    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"
Oct 30 00:40:23.144719 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.144686    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/187932ee-ad15-4170-9e9f-13dc02fa8756-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187932ee-ad15-4170-9e9f-13dc02fa8756-plugins-config-volume\") pod \"187932ee-ad15-4170-9e9f-13dc02fa8756\" (UID: \"187932ee-ad15-4170-9e9f-13dc02fa8756\") "
Oct 30 00:40:23.145201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.144754    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/187932ee-ad15-4170-9e9f-13dc02fa8756-kube-api-access-kp6l2\" (UniqueName: \"kubernetes.io/projected/187932ee-ad15-4170-9e9f-13dc02fa8756-kube-api-access-kp6l2\") pod \"187932ee-ad15-4170-9e9f-13dc02fa8756\" (UID: \"187932ee-ad15-4170-9e9f-13dc02fa8756\") "
Oct 30 00:40:23.145201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.144934    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df19eba-a75b-49a8-9c52-fd7000d43e75-kube-api-access-9bzqb\" (UniqueName: \"kubernetes.io/projected/6df19eba-a75b-49a8-9c52-fd7000d43e75-kube-api-access-9bzqb\") pod \"6df19eba-a75b-49a8-9c52-fd7000d43e75\" (UID: \"6df19eba-a75b-49a8-9c52-fd7000d43e75\") "
Oct 30 00:40:23.145201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.144970    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df19eba-a75b-49a8-9c52-fd7000d43e75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df19eba-a75b-49a8-9c52-fd7000d43e75-plugins-config-volume\") pod \"6df19eba-a75b-49a8-9c52-fd7000d43e75\" (UID: \"6df19eba-a75b-49a8-9c52-fd7000d43e75\") "
Oct 30 00:40:23.145201 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.145163    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187932ee-ad15-4170-9e9f-13dc02fa8756-plugins-config-volume" pod "187932ee-ad15-4170-9e9f-13dc02fa8756" (UID: "187932ee-ad15-4170-9e9f-13dc02fa8756"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:23.145542 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.145519    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df19eba-a75b-49a8-9c52-fd7000d43e75-plugins-config-volume" pod "6df19eba-a75b-49a8-9c52-fd7000d43e75" (UID: "6df19eba-a75b-49a8-9c52-fd7000d43e75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:23.145635 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.145552    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187932ee-ad15-4170-9e9f-13dc02fa8756-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:23.147006 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.146981    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187932ee-ad15-4170-9e9f-13dc02fa8756-kube-api-access-kp6l2" pod "187932ee-ad15-4170-9e9f-13dc02fa8756" (UID: "187932ee-ad15-4170-9e9f-13dc02fa8756"). InnerVolumeSpecName "kube-api-access-kp6l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:40:23.147451 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.147429    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df19eba-a75b-49a8-9c52-fd7000d43e75-kube-api-access-9bzqb" pod "6df19eba-a75b-49a8-9c52-fd7000d43e75" (UID: "6df19eba-a75b-49a8-9c52-fd7000d43e75"). InnerVolumeSpecName "kube-api-access-9bzqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:40:23.246858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.246832    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bzqb\" (UniqueName: \"kubernetes.io/projected/6df19eba-a75b-49a8-9c52-fd7000d43e75-kube-api-access-9bzqb\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:23.246858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.246857    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df19eba-a75b-49a8-9c52-fd7000d43e75-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:23.247052 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.246868    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp6l2\" (UniqueName: \"kubernetes.io/projected/187932ee-ad15-4170-9e9f-13dc02fa8756-kube-api-access-kp6l2\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:23.650465 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.650434    4441 generic.go:356] "Generic (PLEG): container finished" podID="6df19eba-a75b-49a8-9c52-fd7000d43e75" containerID="d7899ddfce98043e050200e85ec196631fd0110d46fbbb21e4becb4fcc08b1ee" exitCode=0
Oct 30 00:40:23.650679 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.650491    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"
Oct 30 00:40:23.650679 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.650508    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg" event={"ID":"6df19eba-a75b-49a8-9c52-fd7000d43e75","Type":"ContainerDied","Data":"d7899ddfce98043e050200e85ec196631fd0110d46fbbb21e4becb4fcc08b1ee"}
Oct 30 00:40:23.650679 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.650568    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg" event={"ID":"6df19eba-a75b-49a8-9c52-fd7000d43e75","Type":"ContainerDied","Data":"c7585a6989e3da733b8798c1169f830b3970bc0f4f606c80d82ed19f2c38080b"}
Oct 30 00:40:23.650679 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.650599    4441 scope.go:122] "RemoveContainer" containerID="d7899ddfce98043e050200e85ec196631fd0110d46fbbb21e4becb4fcc08b1ee"
Oct 30 00:40:23.652608 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.652581    4441 generic.go:356] "Generic (PLEG): container finished" podID="187932ee-ad15-4170-9e9f-13dc02fa8756" containerID="c8e1b4083554026cd0d7626531434c00e80ecaa08a0ac3a220409e960cf30a59" exitCode=0
Oct 30 00:40:23.652732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.652638    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"
Oct 30 00:40:23.652732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.652664    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872" event={"ID":"187932ee-ad15-4170-9e9f-13dc02fa8756","Type":"ContainerDied","Data":"c8e1b4083554026cd0d7626531434c00e80ecaa08a0ac3a220409e960cf30a59"}
Oct 30 00:40:23.652732 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.652726    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872" event={"ID":"187932ee-ad15-4170-9e9f-13dc02fa8756","Type":"ContainerDied","Data":"9d96bedbf43d02e63ca3d2c1e7b0e7196dae5880ca1a04df1420a8b0b64bdabb"}
Oct 30 00:40:23.661773 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.661755    4441 scope.go:122] "RemoveContainer" containerID="d7899ddfce98043e050200e85ec196631fd0110d46fbbb21e4becb4fcc08b1ee"
Oct 30 00:40:23.662061 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:40:23.662020    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7899ddfce98043e050200e85ec196631fd0110d46fbbb21e4becb4fcc08b1ee\": container with ID starting with d7899ddfce98043e050200e85ec196631fd0110d46fbbb21e4becb4fcc08b1ee not found: ID does not exist" containerID="d7899ddfce98043e050200e85ec196631fd0110d46fbbb21e4becb4fcc08b1ee"
Oct 30 00:40:23.662159 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.662069    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7899ddfce98043e050200e85ec196631fd0110d46fbbb21e4becb4fcc08b1ee"} err="failed to get container status \"d7899ddfce98043e050200e85ec196631fd0110d46fbbb21e4becb4fcc08b1ee\": rpc error: code = NotFound desc = could not find container \"d7899ddfce98043e050200e85ec196631fd0110d46fbbb21e4becb4fcc08b1ee\": container with ID starting with d7899ddfce98043e050200e85ec196631fd0110d46fbbb21e4becb4fcc08b1ee not found: ID does not exist"
Oct 30 00:40:23.662159 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.662090    4441 scope.go:122] "RemoveContainer" containerID="c8e1b4083554026cd0d7626531434c00e80ecaa08a0ac3a220409e960cf30a59"
Oct 30 00:40:23.671925 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.671902    4441 scope.go:122] "RemoveContainer" containerID="c8e1b4083554026cd0d7626531434c00e80ecaa08a0ac3a220409e960cf30a59"
Oct 30 00:40:23.672217 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:40:23.672198    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e1b4083554026cd0d7626531434c00e80ecaa08a0ac3a220409e960cf30a59\": container with ID starting with c8e1b4083554026cd0d7626531434c00e80ecaa08a0ac3a220409e960cf30a59 not found: ID does not exist" containerID="c8e1b4083554026cd0d7626531434c00e80ecaa08a0ac3a220409e960cf30a59"
Oct 30 00:40:23.672294 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.672228    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e1b4083554026cd0d7626531434c00e80ecaa08a0ac3a220409e960cf30a59"} err="failed to get container status \"c8e1b4083554026cd0d7626531434c00e80ecaa08a0ac3a220409e960cf30a59\": rpc error: code = NotFound desc = could not find container \"c8e1b4083554026cd0d7626531434c00e80ecaa08a0ac3a220409e960cf30a59\": container with ID starting with c8e1b4083554026cd0d7626531434c00e80ecaa08a0ac3a220409e960cf30a59 not found: ID does not exist"
Oct 30 00:40:23.702816 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.702791    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"]
Oct 30 00:40:23.713852 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.713830    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-63e288-cc6d97cff-c68lg"]
Oct 30 00:40:23.751138 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.751105    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"]
Oct 30 00:40:23.755359 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:23.755339    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-63e288-58fcd47bd8-wq872"]
Oct 30 00:40:24.607697 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:24.607668    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="187932ee-ad15-4170-9e9f-13dc02fa8756" path="/var/lib/kubelet/pods/187932ee-ad15-4170-9e9f-13dc02fa8756/volumes"
Oct 30 00:40:24.608095 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:24.608046    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df19eba-a75b-49a8-9c52-fd7000d43e75" path="/var/lib/kubelet/pods/6df19eba-a75b-49a8-9c52-fd7000d43e75/volumes"
Oct 30 00:40:26.823046 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.822985    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"]
Oct 30 00:40:26.823453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823427    4441 cpu_manager.go:414] "containerMap: removing container" podUID="d4141111-103c-47f4-ac95-1b0d19fbd5cd" containerName="istio-proxy"
Oct 30 00:40:26.823453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823440    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4141111-103c-47f4-ac95-1b0d19fbd5cd" containerName="istio-proxy"
Oct 30 00:40:26.823453 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823451    4441 cpu_manager.go:414] "containerMap: removing container" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" containerName="istio-proxy"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823456    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" containerName="istio-proxy"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823464    4441 cpu_manager.go:414] "containerMap: removing container" podUID="e14e3e15-a207-4782-9905-befa94ac0ddc" containerName="istio-proxy"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823469    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14e3e15-a207-4782-9905-befa94ac0ddc" containerName="istio-proxy"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823479    4441 cpu_manager.go:414] "containerMap: removing container" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823484    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823491    4441 cpu_manager.go:414] "containerMap: removing container" podUID="b439da36-a99a-4847-9884-1c19d9d0b8e8" containerName="maas-api"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823496    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b439da36-a99a-4847-9884-1c19d9d0b8e8" containerName="maas-api"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823502    4441 cpu_manager.go:414] "containerMap: removing container" podUID="6606ae96-4a55-400c-8108-fd38d17c5529" containerName="istio-proxy"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823507    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6606ae96-4a55-400c-8108-fd38d17c5529" containerName="istio-proxy"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823514    4441 cpu_manager.go:414] "containerMap: removing container" podUID="1d5c887e-c6d9-41b0-8b98-78428e123af5" containerName="maas-api"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823518    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d5c887e-c6d9-41b0-8b98-78428e123af5" containerName="maas-api"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823526    4441 cpu_manager.go:414] "containerMap: removing container" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823530    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f550b5f-f099-42d9-bd19-66ec69fce6d5" containerName="istio-proxy"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823536    4441 cpu_manager.go:414] "containerMap: removing container" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823540    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823547    4441 cpu_manager.go:414] "containerMap: removing container" podUID="a3ba27e0-c510-41a6-97ec-83d0c5b637ec" containerName="istio-proxy"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823551    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ba27e0-c510-41a6-97ec-83d0c5b637ec" containerName="istio-proxy"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823558    4441 cpu_manager.go:414] "containerMap: removing container" podUID="fe79b3ad-1c24-4dde-bc30-23a4264e35f9" containerName="istio-proxy"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823562    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe79b3ad-1c24-4dde-bc30-23a4264e35f9" containerName="istio-proxy"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823568    4441 cpu_manager.go:414] "containerMap: removing container" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823573    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823580    4441 cpu_manager.go:414] "containerMap: removing container" podUID="0e2143da-e83f-4fad-a3fe-7509571c484e" containerName="maas-api"
Oct 30 00:40:26.823581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823585    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2143da-e83f-4fad-a3fe-7509571c484e" containerName="maas-api"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823595    4441 cpu_manager.go:414] "containerMap: removing container" podUID="e27789fe-70fb-4912-8873-aabb7694db35" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823600    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27789fe-70fb-4912-8873-aabb7694db35" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823609    4441 cpu_manager.go:414] "containerMap: removing container" podUID="13e3cf7a-4a8c-476e-93ec-2a1734dad886" containerName="payload-processing"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823613    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e3cf7a-4a8c-476e-93ec-2a1734dad886" containerName="payload-processing"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823619    4441 cpu_manager.go:414] "containerMap: removing container" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823623    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823629    4441 cpu_manager.go:414] "containerMap: removing container" podUID="6df19eba-a75b-49a8-9c52-fd7000d43e75" containerName="payload-pre-processing"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823635    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df19eba-a75b-49a8-9c52-fd7000d43e75" containerName="payload-pre-processing"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823650    4441 cpu_manager.go:414] "containerMap: removing container" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823654    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823660    4441 cpu_manager.go:414] "containerMap: removing container" podUID="187932ee-ad15-4170-9e9f-13dc02fa8756" containerName="payload-processing"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823664    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187932ee-ad15-4170-9e9f-13dc02fa8756" containerName="payload-processing"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823670    4441 cpu_manager.go:414] "containerMap: removing container" podUID="d59de270-9763-49fa-8efc-625d58575e48" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823674    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59de270-9763-49fa-8efc-625d58575e48" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823681    4441 cpu_manager.go:414] "containerMap: removing container" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823685    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823696    4441 cpu_manager.go:414] "containerMap: removing container" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823700    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823748    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d43eef-6285-4674-919c-20246cf8c597" containerName="main"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823755    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ba27e0-c510-41a6-97ec-83d0c5b637ec" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823761    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe79b3ad-1c24-4dde-bc30-23a4264e35f9" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823766    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce95d81-779e-4c13-ab4b-04141c618347" containerName="main"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823771    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e2143da-e83f-4fad-a3fe-7509571c484e" containerName="maas-api"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823778    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27789fe-70fb-4912-8873-aabb7694db35" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823783    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e3cf7a-4a8c-476e-93ec-2a1734dad886" containerName="payload-processing"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823789    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="929cb9ab-721f-4d60-b304-50aa1bab3573" containerName="main"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823793    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df19eba-a75b-49a8-9c52-fd7000d43e75" containerName="payload-pre-processing"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823801    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823806    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="187932ee-ad15-4170-9e9f-13dc02fa8756" containerName="payload-processing"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823812    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59de270-9763-49fa-8efc-625d58575e48" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823817    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823823    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="377ffe8e-cc09-4c8b-a25d-c2e486236de5" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823829    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4141111-103c-47f4-ac95-1b0d19fbd5cd" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823834    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0fec19-a16c-40a1-8840-66022d000e75" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823839    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14e3e15-a207-4782-9905-befa94ac0ddc" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823844    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7846274a-6b3c-41c5-ab26-45d2d75bc34f" containerName="main"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823849    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b439da36-a99a-4847-9884-1c19d9d0b8e8" containerName="maas-api"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823854    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6606ae96-4a55-400c-8108-fd38d17c5529" containerName="istio-proxy"
Oct 30 00:40:26.824338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.823858    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d5c887e-c6d9-41b0-8b98-78428e123af5" containerName="maas-api"
Oct 30 00:40:26.828207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.828190    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"
Oct 30 00:40:26.846079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.846056    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"]
Oct 30 00:40:26.867067 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.867045    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:40:26.867262 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.867175    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:40:26.878659 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.878636    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:40:26.878767 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.878692    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\""
Oct 30 00:40:26.986406 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.986380    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp\" (UID: \"541ee940-2c44-4e93-8b85-79a7e9a955a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"
Oct 30 00:40:26.986532 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.986447    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v2ps\" (UniqueName: \"kubernetes.io/projected/541ee940-2c44-4e93-8b85-79a7e9a955a4-kube-api-access-5v2ps\") pod \"maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp\" (UID: \"541ee940-2c44-4e93-8b85-79a7e9a955a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"
Oct 30 00:40:26.986642 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:26.986618    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp\" (UID: \"541ee940-2c44-4e93-8b85-79a7e9a955a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"
Oct 30 00:40:27.088143 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:27.088045    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp\" (UID: \"541ee940-2c44-4e93-8b85-79a7e9a955a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"
Oct 30 00:40:27.088314 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:27.088153    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp\" (UID: \"541ee940-2c44-4e93-8b85-79a7e9a955a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"
Oct 30 00:40:27.088314 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:27.088212    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v2ps\" (UniqueName: \"kubernetes.io/projected/541ee940-2c44-4e93-8b85-79a7e9a955a4-kube-api-access-5v2ps\") pod \"maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp\" (UID: \"541ee940-2c44-4e93-8b85-79a7e9a955a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"
Oct 30 00:40:27.088803 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:27.088770    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp\" (UID: \"541ee940-2c44-4e93-8b85-79a7e9a955a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"
Oct 30 00:40:27.090615 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:27.090596    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp\" (UID: \"541ee940-2c44-4e93-8b85-79a7e9a955a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"
Oct 30 00:40:27.110299 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:27.110276    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v2ps\" (UniqueName: \"kubernetes.io/projected/541ee940-2c44-4e93-8b85-79a7e9a955a4-kube-api-access-5v2ps\") pod \"maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp\" (UID: \"541ee940-2c44-4e93-8b85-79a7e9a955a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"
Oct 30 00:40:27.141164 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:27.141136    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"
Oct 30 00:40:27.440196 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:27.440164    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"]
Oct 30 00:40:27.441424 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:40:27.441392    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541ee940_2c44_4e93_8b85_79a7e9a955a4.slice/crio-f0df08a19b54dd645737b629b2744cf348f08f2cfe0dd1f1a0bc505165e42861 WatchSource:0}: Error finding container f0df08a19b54dd645737b629b2744cf348f08f2cfe0dd1f1a0bc505165e42861: Status 404 returned error can't find the container with id f0df08a19b54dd645737b629b2744cf348f08f2cfe0dd1f1a0bc505165e42861
Oct 30 00:40:27.676960 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:27.676929    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp" event={"ID":"541ee940-2c44-4e93-8b85-79a7e9a955a4","Type":"ContainerStarted","Data":"f0df08a19b54dd645737b629b2744cf348f08f2cfe0dd1f1a0bc505165e42861"}
Oct 30 00:40:28.683408 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:28.683331    4441 generic.go:356] "Generic (PLEG): container finished" podID="541ee940-2c44-4e93-8b85-79a7e9a955a4" containerID="2e0784b3af102ec3db4a63f6c5f621ba1cd9e88126d458c9b53bd046bec2a5bb" exitCode=0
Oct 30 00:40:28.683766 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:28.683418    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp" event={"ID":"541ee940-2c44-4e93-8b85-79a7e9a955a4","Type":"ContainerDied","Data":"2e0784b3af102ec3db4a63f6c5f621ba1cd9e88126d458c9b53bd046bec2a5bb"}
Oct 30 00:40:29.824058 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:29.824021    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"
Oct 30 00:40:29.918459 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:29.918430    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-openshift-ca\") pod \"541ee940-2c44-4e93-8b85-79a7e9a955a4\" (UID: \"541ee940-2c44-4e93-8b85-79a7e9a955a4\") "
Oct 30 00:40:29.918651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:29.918501    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-xks-ca\") pod \"541ee940-2c44-4e93-8b85-79a7e9a955a4\" (UID: \"541ee940-2c44-4e93-8b85-79a7e9a955a4\") "
Oct 30 00:40:29.918651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:29.918563    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541ee940-2c44-4e93-8b85-79a7e9a955a4-kube-api-access-5v2ps\" (UniqueName: \"kubernetes.io/projected/541ee940-2c44-4e93-8b85-79a7e9a955a4-kube-api-access-5v2ps\") pod \"541ee940-2c44-4e93-8b85-79a7e9a955a4\" (UID: \"541ee940-2c44-4e93-8b85-79a7e9a955a4\") "
Oct 30 00:40:29.918835 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:29.918793    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-openshift-ca" pod "541ee940-2c44-4e93-8b85-79a7e9a955a4" (UID: "541ee940-2c44-4e93-8b85-79a7e9a955a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:29.918973 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:29.918957    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-openshift-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:29.920597 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:29.920576    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-xks-ca" pod "541ee940-2c44-4e93-8b85-79a7e9a955a4" (UID: "541ee940-2c44-4e93-8b85-79a7e9a955a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:29.920684 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:29.920670    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541ee940-2c44-4e93-8b85-79a7e9a955a4-kube-api-access-5v2ps" pod "541ee940-2c44-4e93-8b85-79a7e9a955a4" (UID: "541ee940-2c44-4e93-8b85-79a7e9a955a4"). InnerVolumeSpecName "kube-api-access-5v2ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:40:30.020233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:30.020207    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541ee940-2c44-4e93-8b85-79a7e9a955a4-maas-api-xks-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:30.020233 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:30.020232    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v2ps\" (UniqueName: \"kubernetes.io/projected/541ee940-2c44-4e93-8b85-79a7e9a955a4-kube-api-access-5v2ps\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:30.694853 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:30.694821    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"
Oct 30 00:40:30.694853 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:30.694847    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp" event={"ID":"541ee940-2c44-4e93-8b85-79a7e9a955a4","Type":"ContainerDied","Data":"f0df08a19b54dd645737b629b2744cf348f08f2cfe0dd1f1a0bc505165e42861"}
Oct 30 00:40:30.695070 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:30.694893    4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0df08a19b54dd645737b629b2744cf348f08f2cfe0dd1f1a0bc505165e42861"
Oct 30 00:40:31.569500 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:31.569463    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"]
Oct 30 00:40:32.351272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:32.351235    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j" podUID="2af359bf-d670-4f66-9dab-2a95b328705e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:40:33.899381 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:40:33.899338    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:40:37.288879 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:37.288837    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"]
Oct 30 00:40:38.715310 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.715277    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"]
Oct 30 00:40:38.716079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.716057    4441 cpu_manager.go:414] "containerMap: removing container" podUID="541ee940-2c44-4e93-8b85-79a7e9a955a4" containerName="revoke-keys"
Oct 30 00:40:38.716079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.716079    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541ee940-2c44-4e93-8b85-79a7e9a955a4" containerName="revoke-keys"
Oct 30 00:40:38.716244 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.716213    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="541ee940-2c44-4e93-8b85-79a7e9a955a4" containerName="revoke-keys"
Oct 30 00:40:38.719338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.719318    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"
Oct 30 00:40:38.744818 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.744798    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\""
Oct 30 00:40:38.746817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.746794    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"]
Oct 30 00:40:38.804429 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.804397    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4fms\" (UniqueName: \"kubernetes.io/projected/a3a9a063-6cf0-4702-a459-00426fddac61-kube-api-access-p4fms\") pod \"maas-api-revoke-sub-ff446a16-nvtb2\" (UID: \"a3a9a063-6cf0-4702-a459-00426fddac61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"
Oct 30 00:40:38.804548 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.804464    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff446a16-nvtb2\" (UID: \"a3a9a063-6cf0-4702-a459-00426fddac61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"
Oct 30 00:40:38.804649 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.804626    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff446a16-nvtb2\" (UID: \"a3a9a063-6cf0-4702-a459-00426fddac61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"
Oct 30 00:40:38.822140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.822120    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:40:38.822140 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.822120    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:40:38.822265 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.822160    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:40:38.905630 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.905607    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4fms\" (UniqueName: \"kubernetes.io/projected/a3a9a063-6cf0-4702-a459-00426fddac61-kube-api-access-p4fms\") pod \"maas-api-revoke-sub-ff446a16-nvtb2\" (UID: \"a3a9a063-6cf0-4702-a459-00426fddac61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"
Oct 30 00:40:38.905733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.905651    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff446a16-nvtb2\" (UID: \"a3a9a063-6cf0-4702-a459-00426fddac61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"
Oct 30 00:40:38.905733 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.905716    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff446a16-nvtb2\" (UID: \"a3a9a063-6cf0-4702-a459-00426fddac61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"
Oct 30 00:40:38.906331 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.906306    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff446a16-nvtb2\" (UID: \"a3a9a063-6cf0-4702-a459-00426fddac61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"
Oct 30 00:40:38.907966 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.907951    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff446a16-nvtb2\" (UID: \"a3a9a063-6cf0-4702-a459-00426fddac61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"
Oct 30 00:40:38.927999 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:38.927978    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4fms\" (UniqueName: \"kubernetes.io/projected/a3a9a063-6cf0-4702-a459-00426fddac61-kube-api-access-p4fms\") pod \"maas-api-revoke-sub-ff446a16-nvtb2\" (UID: \"a3a9a063-6cf0-4702-a459-00426fddac61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"
Oct 30 00:40:39.033346 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:39.033324    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"
Oct 30 00:40:39.328347 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:39.328116    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"]
Oct 30 00:40:39.330537 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:40:39.330512    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a9a063_6cf0_4702_a459_00426fddac61.slice/crio-76915b7bb9e95ba6b7e55672d97afa8aa064984b52acd55ca99e896708e05f73 WatchSource:0}: Error finding container 76915b7bb9e95ba6b7e55672d97afa8aa064984b52acd55ca99e896708e05f73: Status 404 returned error can't find the container with id 76915b7bb9e95ba6b7e55672d97afa8aa064984b52acd55ca99e896708e05f73
Oct 30 00:40:39.739600 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:39.739567    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2" event={"ID":"a3a9a063-6cf0-4702-a459-00426fddac61","Type":"ContainerStarted","Data":"49eca99200a5f38e5043d0cdac0b3e8a88e48541c9aed999c7eda55765f9e3b2"}
Oct 30 00:40:39.740014 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:39.739650    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2" event={"ID":"a3a9a063-6cf0-4702-a459-00426fddac61","Type":"ContainerStarted","Data":"76915b7bb9e95ba6b7e55672d97afa8aa064984b52acd55ca99e896708e05f73"}
Oct 30 00:40:39.777779 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:39.777734    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2" podStartSLOduration=1.7777223370000002 podStartE2EDuration="1.777722337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:39.776829775 +0000 UTC m=+2149.302123666" watchObservedRunningTime="2026-10-30 00:40:39.777722337 +0000 UTC m=+2149.303016214"
Oct 30 00:40:40.745596 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:40.745561    4441 generic.go:356] "Generic (PLEG): container finished" podID="a3a9a063-6cf0-4702-a459-00426fddac61" containerID="49eca99200a5f38e5043d0cdac0b3e8a88e48541c9aed999c7eda55765f9e3b2" exitCode=0
Oct 30 00:40:40.745996 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:40.745654    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2" event={"ID":"a3a9a063-6cf0-4702-a459-00426fddac61","Type":"ContainerDied","Data":"49eca99200a5f38e5043d0cdac0b3e8a88e48541c9aed999c7eda55765f9e3b2"}
Oct 30 00:40:41.898261 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:41.898238    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"
Oct 30 00:40:41.933353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:41.933328    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-xks-ca\") pod \"a3a9a063-6cf0-4702-a459-00426fddac61\" (UID: \"a3a9a063-6cf0-4702-a459-00426fddac61\") "
Oct 30 00:40:41.933458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:41.933378    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a9a063-6cf0-4702-a459-00426fddac61-kube-api-access-p4fms\" (UniqueName: \"kubernetes.io/projected/a3a9a063-6cf0-4702-a459-00426fddac61-kube-api-access-p4fms\") pod \"a3a9a063-6cf0-4702-a459-00426fddac61\" (UID: \"a3a9a063-6cf0-4702-a459-00426fddac61\") "
Oct 30 00:40:41.933458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:41.933410    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-openshift-ca\") pod \"a3a9a063-6cf0-4702-a459-00426fddac61\" (UID: \"a3a9a063-6cf0-4702-a459-00426fddac61\") "
Oct 30 00:40:41.933826 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:41.933797    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-openshift-ca" pod "a3a9a063-6cf0-4702-a459-00426fddac61" (UID: "a3a9a063-6cf0-4702-a459-00426fddac61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:41.935359 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:41.935335    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-xks-ca" pod "a3a9a063-6cf0-4702-a459-00426fddac61" (UID: "a3a9a063-6cf0-4702-a459-00426fddac61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:41.935516 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:41.935470    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a9a063-6cf0-4702-a459-00426fddac61-kube-api-access-p4fms" pod "a3a9a063-6cf0-4702-a459-00426fddac61" (UID: "a3a9a063-6cf0-4702-a459-00426fddac61"). InnerVolumeSpecName "kube-api-access-p4fms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:40:42.034652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.034577    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-xks-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:42.034652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.034607    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4fms\" (UniqueName: \"kubernetes.io/projected/a3a9a063-6cf0-4702-a459-00426fddac61-kube-api-access-p4fms\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:42.034652 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.034620    4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a9a063-6cf0-4702-a459-00426fddac61-maas-api-openshift-ca\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:42.289529 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.289430    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh" podUID="35394583-9388-440a-8cc8-e0170d541e36" containerName="payload-pre-processing" containerID="cri-o://e6f6fb96bd259102e37dd8dc3c8d1bce54c357c133206441f0ac4dbcacebdf10" gracePeriod=25
Oct 30 00:40:42.333143 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.333119    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"
Oct 30 00:40:42.438458 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.438427    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af359bf-d670-4f66-9dab-2a95b328705e-kube-api-access-qnw62\" (UniqueName: \"kubernetes.io/projected/2af359bf-d670-4f66-9dab-2a95b328705e-kube-api-access-qnw62\") pod \"2af359bf-d670-4f66-9dab-2a95b328705e\" (UID: \"2af359bf-d670-4f66-9dab-2a95b328705e\") "
Oct 30 00:40:42.438651 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.438482    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2af359bf-d670-4f66-9dab-2a95b328705e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af359bf-d670-4f66-9dab-2a95b328705e-plugins-config-volume\") pod \"2af359bf-d670-4f66-9dab-2a95b328705e\" (UID: \"2af359bf-d670-4f66-9dab-2a95b328705e\") "
Oct 30 00:40:42.438919 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.438894    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af359bf-d670-4f66-9dab-2a95b328705e-plugins-config-volume" pod "2af359bf-d670-4f66-9dab-2a95b328705e" (UID: "2af359bf-d670-4f66-9dab-2a95b328705e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:42.440551 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.440531    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af359bf-d670-4f66-9dab-2a95b328705e-kube-api-access-qnw62" pod "2af359bf-d670-4f66-9dab-2a95b328705e" (UID: "2af359bf-d670-4f66-9dab-2a95b328705e"). InnerVolumeSpecName "kube-api-access-qnw62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:40:42.539420 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.539392    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnw62\" (UniqueName: \"kubernetes.io/projected/2af359bf-d670-4f66-9dab-2a95b328705e-kube-api-access-qnw62\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:42.539420 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.539416    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af359bf-d670-4f66-9dab-2a95b328705e-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:42.757347 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.757316    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2"
Oct 30 00:40:42.757487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.757346    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff446a16-nvtb2" event={"ID":"a3a9a063-6cf0-4702-a459-00426fddac61","Type":"ContainerDied","Data":"76915b7bb9e95ba6b7e55672d97afa8aa064984b52acd55ca99e896708e05f73"}
Oct 30 00:40:42.757487 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.757394    4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76915b7bb9e95ba6b7e55672d97afa8aa064984b52acd55ca99e896708e05f73"
Oct 30 00:40:42.758815 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.758788    4441 generic.go:356] "Generic (PLEG): container finished" podID="2af359bf-d670-4f66-9dab-2a95b328705e" containerID="1b4abd5ac1e811073f76877a80755345a7c5080e78060a1ec1c95891d48a2e02" exitCode=0
Oct 30 00:40:42.758946 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.758824    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j" event={"ID":"2af359bf-d670-4f66-9dab-2a95b328705e","Type":"ContainerDied","Data":"1b4abd5ac1e811073f76877a80755345a7c5080e78060a1ec1c95891d48a2e02"}
Oct 30 00:40:42.758946 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.758837    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"
Oct 30 00:40:42.758946 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.758867    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j" event={"ID":"2af359bf-d670-4f66-9dab-2a95b328705e","Type":"ContainerDied","Data":"fa9894ac7c95d50fd3b68462b1900dd76a7223f09a5a731bb2a32f0c1469d7bf"}
Oct 30 00:40:42.758946 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.758901    4441 scope.go:122] "RemoveContainer" containerID="1b4abd5ac1e811073f76877a80755345a7c5080e78060a1ec1c95891d48a2e02"
Oct 30 00:40:42.767672 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.767655    4441 scope.go:122] "RemoveContainer" containerID="1b4abd5ac1e811073f76877a80755345a7c5080e78060a1ec1c95891d48a2e02"
Oct 30 00:40:42.767948 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:40:42.767927    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4abd5ac1e811073f76877a80755345a7c5080e78060a1ec1c95891d48a2e02\": container with ID starting with 1b4abd5ac1e811073f76877a80755345a7c5080e78060a1ec1c95891d48a2e02 not found: ID does not exist" containerID="1b4abd5ac1e811073f76877a80755345a7c5080e78060a1ec1c95891d48a2e02"
Oct 30 00:40:42.768017 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.767956    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4abd5ac1e811073f76877a80755345a7c5080e78060a1ec1c95891d48a2e02"} err="failed to get container status \"1b4abd5ac1e811073f76877a80755345a7c5080e78060a1ec1c95891d48a2e02\": rpc error: code = NotFound desc = could not find container \"1b4abd5ac1e811073f76877a80755345a7c5080e78060a1ec1c95891d48a2e02\": container with ID starting with 1b4abd5ac1e811073f76877a80755345a7c5080e78060a1ec1c95891d48a2e02 not found: ID does not exist"
Oct 30 00:40:42.823056 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.823014    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"]
Oct 30 00:40:42.827722 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:42.827698    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f1268c-589d4f7b94-f5f9j"]
Oct 30 00:40:44.606908 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:44.606865    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2af359bf-d670-4f66-9dab-2a95b328705e" path="/var/lib/kubelet/pods/2af359bf-d670-4f66-9dab-2a95b328705e/volumes"
Oct 30 00:40:46.589839 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.589794    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main" containerID="cri-o://7aef5b118d70a80aff45a1283d2e6d2a7e5ad9b35df02112d4ec080a251d8247" gracePeriod=45
Oct 30 00:40:46.790130 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.790094    4441 generic.go:356] "Generic (PLEG): container finished" podID="462c5bbb-db90-4513-a81e-f7514d428449" containerID="7aef5b118d70a80aff45a1283d2e6d2a7e5ad9b35df02112d4ec080a251d8247" exitCode=0
Oct 30 00:40:46.790274 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.790139    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" event={"ID":"462c5bbb-db90-4513-a81e-f7514d428449","Type":"ContainerDied","Data":"7aef5b118d70a80aff45a1283d2e6d2a7e5ad9b35df02112d4ec080a251d8247"}
Oct 30 00:40:46.839402 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.839365    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:40:46.877938 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.877883    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462c5bbb-db90-4513-a81e-f7514d428449-kube-api-access-9gckx\" (UniqueName: \"kubernetes.io/projected/462c5bbb-db90-4513-a81e-f7514d428449-kube-api-access-9gckx\") pod \"462c5bbb-db90-4513-a81e-f7514d428449\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") "
Oct 30 00:40:46.878059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.877937    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-dshm\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-dshm\") pod \"462c5bbb-db90-4513-a81e-f7514d428449\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") "
Oct 30 00:40:46.878059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.877965    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-home\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-home\") pod \"462c5bbb-db90-4513-a81e-f7514d428449\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") "
Oct 30 00:40:46.878059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.877993    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-model-cache\") pod \"462c5bbb-db90-4513-a81e-f7514d428449\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") "
Oct 30 00:40:46.878197 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.878065    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-tmp-dir\") pod \"462c5bbb-db90-4513-a81e-f7514d428449\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") "
Oct 30 00:40:46.878197 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.878104    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/462c5bbb-db90-4513-a81e-f7514d428449-tls-certs\" (UniqueName: \"kubernetes.io/secret/462c5bbb-db90-4513-a81e-f7514d428449-tls-certs\") pod \"462c5bbb-db90-4513-a81e-f7514d428449\" (UID: \"462c5bbb-db90-4513-a81e-f7514d428449\") "
Oct 30 00:40:46.878273 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.878251    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-home" pod "462c5bbb-db90-4513-a81e-f7514d428449" (UID: "462c5bbb-db90-4513-a81e-f7514d428449"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:40:46.878400 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.878353    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-model-cache" pod "462c5bbb-db90-4513-a81e-f7514d428449" (UID: "462c5bbb-db90-4513-a81e-f7514d428449"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:40:46.878468 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.878416    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-tmp-dir" pod "462c5bbb-db90-4513-a81e-f7514d428449" (UID: "462c5bbb-db90-4513-a81e-f7514d428449"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:40:46.878522 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.878502    4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-model-cache\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:46.878577 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.878523    4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-tmp-dir\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:46.878577 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.878536    4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-home\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:46.880419 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.880393    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462c5bbb-db90-4513-a81e-f7514d428449-tls-certs" pod "462c5bbb-db90-4513-a81e-f7514d428449" (UID: "462c5bbb-db90-4513-a81e-f7514d428449"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:46.880527 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.880434    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462c5bbb-db90-4513-a81e-f7514d428449-kube-api-access-9gckx" pod "462c5bbb-db90-4513-a81e-f7514d428449" (UID: "462c5bbb-db90-4513-a81e-f7514d428449"). InnerVolumeSpecName "kube-api-access-9gckx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:40:46.880598 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.880556    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-dshm" pod "462c5bbb-db90-4513-a81e-f7514d428449" (UID: "462c5bbb-db90-4513-a81e-f7514d428449"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:40:46.979208 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.979161    4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462c5bbb-db90-4513-a81e-f7514d428449-tls-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:46.979208 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.979196    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gckx\" (UniqueName: \"kubernetes.io/projected/462c5bbb-db90-4513-a81e-f7514d428449-kube-api-access-9gckx\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:46.979208 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:46.979205    4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462c5bbb-db90-4513-a81e-f7514d428449-dshm\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:40:47.796601 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:47.796574    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"
Oct 30 00:40:47.797054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:47.796607    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8" event={"ID":"462c5bbb-db90-4513-a81e-f7514d428449","Type":"ContainerDied","Data":"7e5a6d9093e2e1b11b41ecb6c36b3614cb96ea5552c9880cf5116d7d091a10cd"}
Oct 30 00:40:47.797054 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:47.796658    4441 scope.go:122] "RemoveContainer" containerID="7aef5b118d70a80aff45a1283d2e6d2a7e5ad9b35df02112d4ec080a251d8247"
Oct 30 00:40:47.845438 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:47.845407    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"]
Oct 30 00:40:47.856616 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:47.856585    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/distinct-2-w6-6c45bb-kserve-6bd588695b-9l9h8"]
Oct 30 00:40:48.607868 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:48.607836    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462c5bbb-db90-4513-a81e-f7514d428449" path="/var/lib/kubelet/pods/462c5bbb-db90-4513-a81e-f7514d428449/volumes"
Oct 30 00:40:51.057583 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:51.057542    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh" podUID="35394583-9388-440a-8cc8-e0170d541e36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:40:51.797618 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:51.797449    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:40:54.400828 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:54.400793    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"]
Oct 30 00:40:54.412482 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:54.412452    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6c45bb-lvfjp"]
Oct 30 00:40:54.608129 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:40:54.608097    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="541ee940-2c44-4e93-8b85-79a7e9a955a4" path="/var/lib/kubelet/pods/541ee940-2c44-4e93-8b85-79a7e9a955a4/volumes"
Oct 30 00:41:01.053619 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:01.053581    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"]
Oct 30 00:41:01.054153 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:01.053945    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" podUID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" containerName="istio-proxy" containerID="cri-o://377977b3643a7c53a2eef8aded0b1afcb85d65c601dc20dc0b6139c5cc0814c9" gracePeriod=30
Oct 30 00:41:01.058066 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:01.058037    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh" podUID="35394583-9388-440a-8cc8-e0170d541e36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:41:01.064976 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:01.064950    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"]
Oct 30 00:41:01.065251 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:01.065215    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" podUID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" containerName="istio-proxy" containerID="cri-o://896c066a342cba7d8ecb39f89394b6a1cb506fba6107a2caaceb2dc30b6ccdef" gracePeriod=30
Oct 30 00:41:01.066692 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:01.066669    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"]
Oct 30 00:41:01.066843 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:01.066824    4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy" containerID="cri-o://2c48f73955d901fd441696d5336c495b257627ddbc324b18e7a442b22c7c8053" gracePeriod=30
Oct 30 00:41:02.194181 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.194146    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-748768979-s8jhn_fa78720d-333c-4f7a-891f-1d6e4f120472/authorino/0.log"
Oct 30 00:41:02.416700 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.416679    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-t2mxs_03e20d41-ae7d-4ea8-8528-12f00023b3b4/manager/0.log"
Oct 30 00:41:02.567626 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.567601    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"
Oct 30 00:41:02.667895 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.667865    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35394583-9388-440a-8cc8-e0170d541e36-kube-api-access-85k9j\" (UniqueName: \"kubernetes.io/projected/35394583-9388-440a-8cc8-e0170d541e36-kube-api-access-85k9j\") pod \"35394583-9388-440a-8cc8-e0170d541e36\" (UID: \"35394583-9388-440a-8cc8-e0170d541e36\") "
Oct 30 00:41:02.668083 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.667932    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35394583-9388-440a-8cc8-e0170d541e36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35394583-9388-440a-8cc8-e0170d541e36-plugins-config-volume\") pod \"35394583-9388-440a-8cc8-e0170d541e36\" (UID: \"35394583-9388-440a-8cc8-e0170d541e36\") "
Oct 30 00:41:02.668424 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.668394    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35394583-9388-440a-8cc8-e0170d541e36-plugins-config-volume" pod "35394583-9388-440a-8cc8-e0170d541e36" (UID: "35394583-9388-440a-8cc8-e0170d541e36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:41:02.670115 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.670089    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35394583-9388-440a-8cc8-e0170d541e36-kube-api-access-85k9j" pod "35394583-9388-440a-8cc8-e0170d541e36" (UID: "35394583-9388-440a-8cc8-e0170d541e36"). InnerVolumeSpecName "kube-api-access-85k9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:41:02.688335 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.688312    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8jsw"]
Oct 30 00:41:02.769378 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.769352    4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35394583-9388-440a-8cc8-e0170d541e36-plugins-config-volume\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:02.769378 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.769374    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85k9j\" (UniqueName: \"kubernetes.io/projected/35394583-9388-440a-8cc8-e0170d541e36-kube-api-access-85k9j\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:02.879896 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.879852    4441 generic.go:356] "Generic (PLEG): container finished" podID="35394583-9388-440a-8cc8-e0170d541e36" containerID="e6f6fb96bd259102e37dd8dc3c8d1bce54c357c133206441f0ac4dbcacebdf10" exitCode=0
Oct 30 00:41:02.880120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.879900    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"
Oct 30 00:41:02.880120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.879944    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh" event={"ID":"35394583-9388-440a-8cc8-e0170d541e36","Type":"ContainerDied","Data":"e6f6fb96bd259102e37dd8dc3c8d1bce54c357c133206441f0ac4dbcacebdf10"}
Oct 30 00:41:02.880120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.880001    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh" event={"ID":"35394583-9388-440a-8cc8-e0170d541e36","Type":"ContainerDied","Data":"f6b7a47d2cd90bd2c87f48d8154d6f4713c3037ca8bb3e7c69a7e1e142fcdcd5"}
Oct 30 00:41:02.880120 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.880047    4441 scope.go:122] "RemoveContainer" containerID="e6f6fb96bd259102e37dd8dc3c8d1bce54c357c133206441f0ac4dbcacebdf10"
Oct 30 00:41:02.890017 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.889996    4441 scope.go:122] "RemoveContainer" containerID="e6f6fb96bd259102e37dd8dc3c8d1bce54c357c133206441f0ac4dbcacebdf10"
Oct 30 00:41:02.890302 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:41:02.890279    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f6fb96bd259102e37dd8dc3c8d1bce54c357c133206441f0ac4dbcacebdf10\": container with ID starting with e6f6fb96bd259102e37dd8dc3c8d1bce54c357c133206441f0ac4dbcacebdf10 not found: ID does not exist" containerID="e6f6fb96bd259102e37dd8dc3c8d1bce54c357c133206441f0ac4dbcacebdf10"
Oct 30 00:41:02.890370 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.890315    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f6fb96bd259102e37dd8dc3c8d1bce54c357c133206441f0ac4dbcacebdf10"} err="failed to get container status \"e6f6fb96bd259102e37dd8dc3c8d1bce54c357c133206441f0ac4dbcacebdf10\": rpc error: code = NotFound desc = could not find container \"e6f6fb96bd259102e37dd8dc3c8d1bce54c357c133206441f0ac4dbcacebdf10\": container with ID starting with e6f6fb96bd259102e37dd8dc3c8d1bce54c357c133206441f0ac4dbcacebdf10 not found: ID does not exist"
Oct 30 00:41:02.929514 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.929484    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"]
Oct 30 00:41:02.935146 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:02.935125    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6c45bb-769575b895-5bqhh"]
Oct 30 00:41:03.213793 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:03.213695    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tmkmd_185b11f8-5a8a-4282-b9ae-53c9182d0668/manager/1.log"
Oct 30 00:41:03.434824 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:03.434794    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-j8jsw_7883212e-1f9e-471c-be5d-51d0f0b20188/limitador/0.log"
Oct 30 00:41:03.901654 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:41:03.901621    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:41:04.206662 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:04.206563    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tmkmd_185b11f8-5a8a-4282-b9ae-53c9182d0668/manager/0.log"
Oct 30 00:41:04.608319 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:04.608284    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35394583-9388-440a-8cc8-e0170d541e36" path="/var/lib/kubelet/pods/35394583-9388-440a-8cc8-e0170d541e36/volumes"
Oct 30 00:41:04.988019 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:04.987990    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/1.log"
Oct 30 00:41:06.249103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.249015    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9lbf8_5ba56bd4-0351-41df-9717-c0ba2ec2d124/manager/0.log"
Oct 30 00:41:06.288886 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.288855    4441 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 00:41:06.289077 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.288926    4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" podUID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 00:41:06.326462 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.326402    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:41:06.376511 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.376486    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:41:06.379156 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.379131    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:41:06.509282 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509253    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istiod-ca-cert\") pod \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") "
Oct 30 00:41:06.509464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509290    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-envoy\") pod \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") "
Oct 30 00:41:06.509464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509313    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-kube-api-access-zxxjn\" (UniqueName: \"kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-kube-api-access-zxxjn\") pod \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") "
Oct 30 00:41:06.509464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509338    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-podinfo\") pod \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") "
Oct 30 00:41:06.509464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509363    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-kube-api-access-gq9m8\" (UniqueName: \"kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-kube-api-access-gq9m8\") pod \"37484057-61d0-44a9-96db-4d814905649b\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") "
Oct 30 00:41:06.509464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509402    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-credential-socket\") pod \"37484057-61d0-44a9-96db-4d814905649b\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") "
Oct 30 00:41:06.509464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509434    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-token\" (UniqueName: \"kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-token\") pod \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") "
Oct 30 00:41:06.509464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509463    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37484057-61d0-44a9-96db-4d814905649b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37484057-61d0-44a9-96db-4d814905649b-istiod-ca-cert\") pod \"37484057-61d0-44a9-96db-4d814905649b\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") "
Oct 30 00:41:06.509865 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509480    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-socket\") pod \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") "
Oct 30 00:41:06.509865 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509501    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-istio-token\" (UniqueName: \"kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-istio-token\") pod \"37484057-61d0-44a9-96db-4d814905649b\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") "
Oct 30 00:41:06.509865 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509528    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-certs\") pod \"37484057-61d0-44a9-96db-4d814905649b\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") "
Oct 30 00:41:06.509865 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509550    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-envoy\") pod \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") "
Oct 30 00:41:06.509865 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509584    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-certs\") pod \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") "
Oct 30 00:41:06.509865 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509613    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-token\" (UniqueName: \"kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-token\") pod \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") "
Oct 30 00:41:06.509865 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509677    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-socket\") pod \"37484057-61d0-44a9-96db-4d814905649b\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") "
Oct 30 00:41:06.509865 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509691    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istiod-ca-cert" pod "16ffa28d-7b18-46c3-99e0-3783316b0dc6" (UID: "16ffa28d-7b18-46c3-99e0-3783316b0dc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:41:06.509865 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509709    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-envoy\") pod \"37484057-61d0-44a9-96db-4d814905649b\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") "
Oct 30 00:41:06.510395 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509866    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-data\") pod \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") "
Oct 30 00:41:06.510395 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509907    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-socket\") pod \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") "
Oct 30 00:41:06.510395 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509937    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-podinfo\") pod \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") "
Oct 30 00:41:06.510395 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.509968    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-data\") pod \"37484057-61d0-44a9-96db-4d814905649b\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") "
Oct 30 00:41:06.510395 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510004    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-kube-api-access-bp474\" (UniqueName: \"kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-kube-api-access-bp474\") pod \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") "
Oct 30 00:41:06.510395 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510061    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-credential-socket\") pod \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") "
Oct 30 00:41:06.510395 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510104    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37484057-61d0-44a9-96db-4d814905649b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37484057-61d0-44a9-96db-4d814905649b-istio-podinfo\") pod \"37484057-61d0-44a9-96db-4d814905649b\" (UID: \"37484057-61d0-44a9-96db-4d814905649b\") "
Oct 30 00:41:06.510395 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510135    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-data\") pod \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") "
Oct 30 00:41:06.510395 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510172    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-credential-socket\") pod \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") "
Oct 30 00:41:06.510395 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510177    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37484057-61d0-44a9-96db-4d814905649b-istiod-ca-cert" pod "37484057-61d0-44a9-96db-4d814905649b" (UID: "37484057-61d0-44a9-96db-4d814905649b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:41:06.510395 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510217    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istiod-ca-cert\") pod \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\" (UID: \"ca2b1b2b-cc9e-4159-a4ab-247692d92831\") "
Oct 30 00:41:06.510395 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510246    4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-certs\") pod \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\" (UID: \"16ffa28d-7b18-46c3-99e0-3783316b0dc6\") "
Oct 30 00:41:06.511050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510405    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-socket" pod "16ffa28d-7b18-46c3-99e0-3783316b0dc6" (UID: "16ffa28d-7b18-46c3-99e0-3783316b0dc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.511050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510492    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-certs" pod "ca2b1b2b-cc9e-4159-a4ab-247692d92831" (UID: "ca2b1b2b-cc9e-4159-a4ab-247692d92831"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.511050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510532    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-credential-socket" pod "37484057-61d0-44a9-96db-4d814905649b" (UID: "37484057-61d0-44a9-96db-4d814905649b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.511050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510722    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-certs" pod "37484057-61d0-44a9-96db-4d814905649b" (UID: "37484057-61d0-44a9-96db-4d814905649b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.511050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510666    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.511050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510884    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-certs" pod "16ffa28d-7b18-46c3-99e0-3783316b0dc6" (UID: "16ffa28d-7b18-46c3-99e0-3783316b0dc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.511050 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.510959    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-socket" pod "37484057-61d0-44a9-96db-4d814905649b" (UID: "37484057-61d0-44a9-96db-4d814905649b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.511534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.511226    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-credential-socket" pod "ca2b1b2b-cc9e-4159-a4ab-247692d92831" (UID: "ca2b1b2b-cc9e-4159-a4ab-247692d92831"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.511534 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.511423    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-socket" pod "ca2b1b2b-cc9e-4159-a4ab-247692d92831" (UID: "ca2b1b2b-cc9e-4159-a4ab-247692d92831"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.512345 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.511772    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-data" pod "37484057-61d0-44a9-96db-4d814905649b" (UID: "37484057-61d0-44a9-96db-4d814905649b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.512345 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.511821    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-data" pod "ca2b1b2b-cc9e-4159-a4ab-247692d92831" (UID: "ca2b1b2b-cc9e-4159-a4ab-247692d92831"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.512345 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.512209    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-data" pod "16ffa28d-7b18-46c3-99e0-3783316b0dc6" (UID: "16ffa28d-7b18-46c3-99e0-3783316b0dc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.512345 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.512275    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-credential-socket" pod "16ffa28d-7b18-46c3-99e0-3783316b0dc6" (UID: "16ffa28d-7b18-46c3-99e0-3783316b0dc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.512636 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.512519    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istiod-ca-cert" pod "ca2b1b2b-cc9e-4159-a4ab-247692d92831" (UID: "ca2b1b2b-cc9e-4159-a4ab-247692d92831"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:41:06.512838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.512813    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-envoy" pod "16ffa28d-7b18-46c3-99e0-3783316b0dc6" (UID: "16ffa28d-7b18-46c3-99e0-3783316b0dc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.512838 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.512824    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-kube-api-access-gq9m8" pod "37484057-61d0-44a9-96db-4d814905649b" (UID: "37484057-61d0-44a9-96db-4d814905649b"). InnerVolumeSpecName "kube-api-access-gq9m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:41:06.512991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.512845    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-podinfo" pod "ca2b1b2b-cc9e-4159-a4ab-247692d92831" (UID: "ca2b1b2b-cc9e-4159-a4ab-247692d92831"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:41:06.512991 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.512966    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-token" pod "ca2b1b2b-cc9e-4159-a4ab-247692d92831" (UID: "ca2b1b2b-cc9e-4159-a4ab-247692d92831"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:41:06.513188 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.513162    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-istio-token" pod "37484057-61d0-44a9-96db-4d814905649b" (UID: "37484057-61d0-44a9-96db-4d814905649b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:41:06.513705 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.513672    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-envoy" pod "37484057-61d0-44a9-96db-4d814905649b" (UID: "37484057-61d0-44a9-96db-4d814905649b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.513943 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.513912    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-podinfo" pod "16ffa28d-7b18-46c3-99e0-3783316b0dc6" (UID: "16ffa28d-7b18-46c3-99e0-3783316b0dc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:41:06.514421 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.514397    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-kube-api-access-zxxjn" pod "ca2b1b2b-cc9e-4159-a4ab-247692d92831" (UID: "ca2b1b2b-cc9e-4159-a4ab-247692d92831"). InnerVolumeSpecName "kube-api-access-zxxjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:41:06.514549 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.514533    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-kube-api-access-bp474" pod "16ffa28d-7b18-46c3-99e0-3783316b0dc6" (UID: "16ffa28d-7b18-46c3-99e0-3783316b0dc6"). InnerVolumeSpecName "kube-api-access-bp474". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:41:06.514648 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.514634    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-token" pod "16ffa28d-7b18-46c3-99e0-3783316b0dc6" (UID: "16ffa28d-7b18-46c3-99e0-3783316b0dc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:41:06.514697 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.514644    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-envoy" pod "ca2b1b2b-cc9e-4159-a4ab-247692d92831" (UID: "ca2b1b2b-cc9e-4159-a4ab-247692d92831"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:06.514833 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.514819    4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37484057-61d0-44a9-96db-4d814905649b-istio-podinfo" pod "37484057-61d0-44a9-96db-4d814905649b" (UID: "37484057-61d0-44a9-96db-4d814905649b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:41:06.611707 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611641    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611707 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611674    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxxjn\" (UniqueName: \"kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-kube-api-access-zxxjn\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611707 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611691    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611707 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611706    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq9m8\" (UniqueName: \"kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-kube-api-access-gq9m8\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611721    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611734    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611746    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37484057-61d0-44a9-96db-4d814905649b-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611758    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611769    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37484057-61d0-44a9-96db-4d814905649b-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611781    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611792    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611802    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611813    4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-token\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611825    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611835    4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-envoy\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611847    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611858    4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-workload-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611869    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611881    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37484057-61d0-44a9-96db-4d814905649b-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611893    4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp474\" (UniqueName: \"kubernetes.io/projected/16ffa28d-7b18-46c3-99e0-3783316b0dc6-kube-api-access-bp474\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611904    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2b1b2b-cc9e-4159-a4ab-247692d92831-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611917    4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37484057-61d0-44a9-96db-4d814905649b-istio-podinfo\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611928    4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-istio-data\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611938    4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-credential-socket\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611953    4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2b1b2b-cc9e-4159-a4ab-247692d92831-istiod-ca-cert\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.611970 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.611966    4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ffa28d-7b18-46c3-99e0-3783316b0dc6-workload-certs\") on node \"ip-10-0-137-69.ec2.internal\" DevicePath \"\""
Oct 30 00:41:06.898817 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.898752    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/0.log"
Oct 30 00:41:06.903079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.903057    4441 generic.go:356] "Generic (PLEG): container finished" podID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" containerID="896c066a342cba7d8ecb39f89394b6a1cb506fba6107a2caaceb2dc30b6ccdef" exitCode=0
Oct 30 00:41:06.903207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.903106    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"
Oct 30 00:41:06.903207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.903128    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" event={"ID":"16ffa28d-7b18-46c3-99e0-3783316b0dc6","Type":"ContainerDied","Data":"896c066a342cba7d8ecb39f89394b6a1cb506fba6107a2caaceb2dc30b6ccdef"}
Oct 30 00:41:06.903207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.903177    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd" event={"ID":"16ffa28d-7b18-46c3-99e0-3783316b0dc6","Type":"ContainerDied","Data":"b976a51639f648adce8a27794a704e956fec589460a601dedca1a2f083afb87b"}
Oct 30 00:41:06.903207 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.903201    4441 scope.go:122] "RemoveContainer" containerID="896c066a342cba7d8ecb39f89394b6a1cb506fba6107a2caaceb2dc30b6ccdef"
Oct 30 00:41:06.904549 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.904529    4441 generic.go:356] "Generic (PLEG): container finished" podID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" containerID="377977b3643a7c53a2eef8aded0b1afcb85d65c601dc20dc0b6139c5cc0814c9" exitCode=0
Oct 30 00:41:06.904657 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.904578    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" event={"ID":"ca2b1b2b-cc9e-4159-a4ab-247692d92831","Type":"ContainerDied","Data":"377977b3643a7c53a2eef8aded0b1afcb85d65c601dc20dc0b6139c5cc0814c9"}
Oct 30 00:41:06.904657 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.904634    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc" event={"ID":"ca2b1b2b-cc9e-4159-a4ab-247692d92831","Type":"ContainerDied","Data":"5bcb9296c6947500418b8195ce6f1ff3554aa5dda82d2705fa84a3fab8f76d8f"}
Oct 30 00:41:06.904657 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.904587    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"
Oct 30 00:41:06.906464 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.906444    4441 generic.go:356] "Generic (PLEG): container finished" podID="37484057-61d0-44a9-96db-4d814905649b" containerID="2c48f73955d901fd441696d5336c495b257627ddbc324b18e7a442b22c7c8053" exitCode=0
Oct 30 00:41:06.906565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.906469    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" event={"ID":"37484057-61d0-44a9-96db-4d814905649b","Type":"ContainerDied","Data":"2c48f73955d901fd441696d5336c495b257627ddbc324b18e7a442b22c7c8053"}
Oct 30 00:41:06.906565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.906501    4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"
Oct 30 00:41:06.906565 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.906518    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps" event={"ID":"37484057-61d0-44a9-96db-4d814905649b","Type":"ContainerDied","Data":"1b7f480b242b0cac038ceb48593980223c2d3708816d0e1757ba0df93c1ebbf6"}
Oct 30 00:41:06.913648 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.913632    4441 scope.go:122] "RemoveContainer" containerID="896c066a342cba7d8ecb39f89394b6a1cb506fba6107a2caaceb2dc30b6ccdef"
Oct 30 00:41:06.913949 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:41:06.913923    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896c066a342cba7d8ecb39f89394b6a1cb506fba6107a2caaceb2dc30b6ccdef\": container with ID starting with 896c066a342cba7d8ecb39f89394b6a1cb506fba6107a2caaceb2dc30b6ccdef not found: ID does not exist" containerID="896c066a342cba7d8ecb39f89394b6a1cb506fba6107a2caaceb2dc30b6ccdef"
Oct 30 00:41:06.914072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.913959    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896c066a342cba7d8ecb39f89394b6a1cb506fba6107a2caaceb2dc30b6ccdef"} err="failed to get container status \"896c066a342cba7d8ecb39f89394b6a1cb506fba6107a2caaceb2dc30b6ccdef\": rpc error: code = NotFound desc = could not find container \"896c066a342cba7d8ecb39f89394b6a1cb506fba6107a2caaceb2dc30b6ccdef\": container with ID starting with 896c066a342cba7d8ecb39f89394b6a1cb506fba6107a2caaceb2dc30b6ccdef not found: ID does not exist"
Oct 30 00:41:06.914072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.913977    4441 scope.go:122] "RemoveContainer" containerID="377977b3643a7c53a2eef8aded0b1afcb85d65c601dc20dc0b6139c5cc0814c9"
Oct 30 00:41:06.922267 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.922249    4441 scope.go:122] "RemoveContainer" containerID="377977b3643a7c53a2eef8aded0b1afcb85d65c601dc20dc0b6139c5cc0814c9"
Oct 30 00:41:06.922528 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:41:06.922511    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377977b3643a7c53a2eef8aded0b1afcb85d65c601dc20dc0b6139c5cc0814c9\": container with ID starting with 377977b3643a7c53a2eef8aded0b1afcb85d65c601dc20dc0b6139c5cc0814c9 not found: ID does not exist" containerID="377977b3643a7c53a2eef8aded0b1afcb85d65c601dc20dc0b6139c5cc0814c9"
Oct 30 00:41:06.922580 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.922535    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377977b3643a7c53a2eef8aded0b1afcb85d65c601dc20dc0b6139c5cc0814c9"} err="failed to get container status \"377977b3643a7c53a2eef8aded0b1afcb85d65c601dc20dc0b6139c5cc0814c9\": rpc error: code = NotFound desc = could not find container \"377977b3643a7c53a2eef8aded0b1afcb85d65c601dc20dc0b6139c5cc0814c9\": container with ID starting with 377977b3643a7c53a2eef8aded0b1afcb85d65c601dc20dc0b6139c5cc0814c9 not found: ID does not exist"
Oct 30 00:41:06.922580 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.922554    4441 scope.go:122] "RemoveContainer" containerID="2c48f73955d901fd441696d5336c495b257627ddbc324b18e7a442b22c7c8053"
Oct 30 00:41:06.930883 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.930862    4441 scope.go:122] "RemoveContainer" containerID="e5666b8df36f3ed97ae9fb8ff8d22b413f166b52c0c196091738be96bfee841b"
Oct 30 00:41:06.939258 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.939235    4441 scope.go:122] "RemoveContainer" containerID="2c48f73955d901fd441696d5336c495b257627ddbc324b18e7a442b22c7c8053"
Oct 30 00:41:06.939502 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:41:06.939481    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c48f73955d901fd441696d5336c495b257627ddbc324b18e7a442b22c7c8053\": container with ID starting with 2c48f73955d901fd441696d5336c495b257627ddbc324b18e7a442b22c7c8053 not found: ID does not exist" containerID="2c48f73955d901fd441696d5336c495b257627ddbc324b18e7a442b22c7c8053"
Oct 30 00:41:06.939591 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.939513    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c48f73955d901fd441696d5336c495b257627ddbc324b18e7a442b22c7c8053"} err="failed to get container status \"2c48f73955d901fd441696d5336c495b257627ddbc324b18e7a442b22c7c8053\": rpc error: code = NotFound desc = could not find container \"2c48f73955d901fd441696d5336c495b257627ddbc324b18e7a442b22c7c8053\": container with ID starting with 2c48f73955d901fd441696d5336c495b257627ddbc324b18e7a442b22c7c8053 not found: ID does not exist"
Oct 30 00:41:06.939591 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.939536    4441 scope.go:122] "RemoveContainer" containerID="e5666b8df36f3ed97ae9fb8ff8d22b413f166b52c0c196091738be96bfee841b"
Oct 30 00:41:06.939763 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:41:06.939745    4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5666b8df36f3ed97ae9fb8ff8d22b413f166b52c0c196091738be96bfee841b\": container with ID starting with e5666b8df36f3ed97ae9fb8ff8d22b413f166b52c0c196091738be96bfee841b not found: ID does not exist" containerID="e5666b8df36f3ed97ae9fb8ff8d22b413f166b52c0c196091738be96bfee841b"
Oct 30 00:41:06.939807 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.939771    4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5666b8df36f3ed97ae9fb8ff8d22b413f166b52c0c196091738be96bfee841b"} err="failed to get container status \"e5666b8df36f3ed97ae9fb8ff8d22b413f166b52c0c196091738be96bfee841b\": rpc error: code = NotFound desc = could not find container \"e5666b8df36f3ed97ae9fb8ff8d22b413f166b52c0c196091738be96bfee841b\": container with ID starting with e5666b8df36f3ed97ae9fb8ff8d22b413f166b52c0c196091738be96bfee841b not found: ID does not exist"
Oct 30 00:41:06.951676 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.951652    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"]
Oct 30 00:41:06.963531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.963507    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-md7hd"]
Oct 30 00:41:06.986580 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.986555    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"]
Oct 30 00:41:06.990389 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:06.990366    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-dhfcc"]
Oct 30 00:41:07.022985 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:07.022962    4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"]
Oct 30 00:41:07.026668 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:07.026647    4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5w9ps"]
Oct 30 00:41:07.658103 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:07.658075    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-748768979-s8jhn_fa78720d-333c-4f7a-891f-1d6e4f120472/authorino/0.log"
Oct 30 00:41:08.607497 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:08.607462    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" path="/var/lib/kubelet/pods/16ffa28d-7b18-46c3-99e0-3783316b0dc6/volumes"
Oct 30 00:41:08.607981 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:08.607962    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37484057-61d0-44a9-96db-4d814905649b" path="/var/lib/kubelet/pods/37484057-61d0-44a9-96db-4d814905649b/volumes"
Oct 30 00:41:08.608505 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:08.608488    4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" path="/var/lib/kubelet/pods/ca2b1b2b-cc9e-4159-a4ab-247692d92831/volumes"
Oct 30 00:41:18.482226 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:18.482193    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/1.log"
Oct 30 00:41:19.745736 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:19.745703    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9lbf8_5ba56bd4-0351-41df-9717-c0ba2ec2d124/manager/0.log"
Oct 30 00:41:20.358826 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:20.358773    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/0.log"
Oct 30 00:41:24.513989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:24.513962    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ff446a16-nvtb2_a3a9a063-6cf0-4702-a459-00426fddac61/revoke-keys/0.log"
Oct 30 00:41:26.167762 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:26.167718    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-748768979-s8jhn_fa78720d-333c-4f7a-891f-1d6e4f120472/authorino/0.log"
Oct 30 00:41:26.377499 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:26.377452    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-t2mxs_03e20d41-ae7d-4ea8-8528-12f00023b3b4/manager/0.log"
Oct 30 00:41:27.047757 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:27.047724    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tmkmd_185b11f8-5a8a-4282-b9ae-53c9182d0668/manager/1.log"
Oct 30 00:41:27.263267 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:27.263241    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-j8jsw_7883212e-1f9e-471c-be5d-51d0f0b20188/limitador/0.log"
Oct 30 00:41:28.015101 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:28.015068    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tmkmd_185b11f8-5a8a-4282-b9ae-53c9182d0668/manager/0.log"
Oct 30 00:41:28.970845 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:28.970816    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l2c22_04666c76-7d71-480d-bae1-27c724358c54/discovery/0.log"
Oct 30 00:41:29.350418 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:29.350394    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d88dd7bf-bblqz_36ed0996-1da5-471c-a110-fb73de534126/kube-auth-proxy/0.log"
Oct 30 00:41:30.102165 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:30.102137    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-mjckv_972bef50-60cd-4b01-85ab-c17359d1327b/istio-proxy/0.log"
Oct 30 00:41:32.355918 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:32.355890    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-external-token-limit-simulated-kserve-65fc98cd9c-vkn7l_a70f3491-6506-456f-ab45-90b1291f086d/main/0.log"
Oct 30 00:41:32.732939 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:32.732907    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qm4r_a40101d4-2789-4fa1-b9cf-6fd2e518efe5/main/0.log"
Oct 30 00:41:33.092064 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:33.091979    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-hx728_7100f212-f2df-48aa-9895-d51333bc10ad/main/0.log"
Oct 30 00:41:33.897385 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:41:33.897345    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:41:48.968072 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:48.968040    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v5tjs_3d12dbae-7cc9-41fd-af0d-3dc152a3d6f5/global-pull-secret-syncer/0.log"
Oct 30 00:41:49.836708 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:49.836679    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kmg9c_bf392012-ba7c-4999-b0fe-f0ae7cee60f7/konnectivity-agent/0.log"
Oct 30 00:41:50.111889 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:50.111803    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-69.ec2.internal_e3314e96d9e96380a527867d39d2fde5/haproxy/0.log"
Oct 30 00:41:51.313114 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:51.313082    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/1.log"
Oct 30 00:41:51.313602 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:51.313427    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/0.log"
Oct 30 00:41:53.973484 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:53.973448    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9lbf8_5ba56bd4-0351-41df-9717-c0ba2ec2d124/manager/0.log"
Oct 30 00:41:55.306763 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:55.306731    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/1.log"
Oct 30 00:41:55.307230 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:55.307109    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/0.log"
Oct 30 00:41:56.986182 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:56.986129    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-748768979-s8jhn_fa78720d-333c-4f7a-891f-1d6e4f120472/authorino/0.log"
Oct 30 00:41:57.085460 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:57.085416    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-t2mxs_03e20d41-ae7d-4ea8-8528-12f00023b3b4/manager/0.log"
Oct 30 00:41:57.613055 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:57.613001    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tmkmd_185b11f8-5a8a-4282-b9ae-53c9182d0668/manager/1.log"
Oct 30 00:41:57.746002 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:57.745972    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9lbf8_5ba56bd4-0351-41df-9717-c0ba2ec2d124/manager/0.log"
Oct 30 00:41:57.761678 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:57.761654    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tmkmd_185b11f8-5a8a-4282-b9ae-53c9182d0668/manager/0.log"
Oct 30 00:41:57.928680 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:57.928616    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-j8jsw_7883212e-1f9e-471c-be5d-51d0f0b20188/limitador/0.log"
Oct 30 00:41:58.551452 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:58.551389    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/1.log"
Oct 30 00:41:58.551867 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:58.551582    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/0.log"
Oct 30 00:41:59.099211 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:59.099182    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/1.log"
Oct 30 00:41:59.099623 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:41:59.099603    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/0.log"
Oct 30 00:42:00.114947 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.114901    4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"]
Oct 30 00:42:00.115681 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115656    4441 cpu_manager.go:414] "containerMap: removing container" podUID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" containerName="istio-proxy"
Oct 30 00:42:00.115796 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115683    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" containerName="istio-proxy"
Oct 30 00:42:00.115796 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115712    4441 cpu_manager.go:414] "containerMap: removing container" podUID="a3a9a063-6cf0-4702-a459-00426fddac61" containerName="revoke-keys"
Oct 30 00:42:00.115796 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115719    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a9a063-6cf0-4702-a459-00426fddac61" containerName="revoke-keys"
Oct 30 00:42:00.115796 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115752    4441 cpu_manager.go:414] "containerMap: removing container" podUID="35394583-9388-440a-8cc8-e0170d541e36" containerName="payload-pre-processing"
Oct 30 00:42:00.115796 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115764    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35394583-9388-440a-8cc8-e0170d541e36" containerName="payload-pre-processing"
Oct 30 00:42:00.116086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115802    4441 cpu_manager.go:414] "containerMap: removing container" podUID="2af359bf-d670-4f66-9dab-2a95b328705e" containerName="payload-processing"
Oct 30 00:42:00.116086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115809    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af359bf-d670-4f66-9dab-2a95b328705e" containerName="payload-processing"
Oct 30 00:42:00.116086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115825    4441 cpu_manager.go:414] "containerMap: removing container" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy"
Oct 30 00:42:00.116086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115834    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy"
Oct 30 00:42:00.116086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115852    4441 cpu_manager.go:414] "containerMap: removing container" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy"
Oct 30 00:42:00.116086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115858    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy"
Oct 30 00:42:00.116086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115869    4441 cpu_manager.go:414] "containerMap: removing container" podUID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" containerName="istio-proxy"
Oct 30 00:42:00.116086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115876    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" containerName="istio-proxy"
Oct 30 00:42:00.116086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115903    4441 cpu_manager.go:414] "containerMap: removing container" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main"
Oct 30 00:42:00.116086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.115910    4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main"
Oct 30 00:42:00.116086 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.116076    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="462c5bbb-db90-4513-a81e-f7514d428449" containerName="main"
Oct 30 00:42:00.116607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.116097    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2b1b2b-cc9e-4159-a4ab-247692d92831" containerName="istio-proxy"
Oct 30 00:42:00.116607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.116114    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a9a063-6cf0-4702-a459-00426fddac61" containerName="revoke-keys"
Oct 30 00:42:00.116607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.116126    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="35394583-9388-440a-8cc8-e0170d541e36" containerName="payload-pre-processing"
Oct 30 00:42:00.116607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.116146    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af359bf-d670-4f66-9dab-2a95b328705e" containerName="payload-processing"
Oct 30 00:42:00.116607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.116161    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy"
Oct 30 00:42:00.116607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.116170    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="37484057-61d0-44a9-96db-4d814905649b" containerName="istio-proxy"
Oct 30 00:42:00.116607 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.116187    4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ffa28d-7b18-46c3-99e0-3783316b0dc6" containerName="istio-proxy"
Oct 30 00:42:00.123189 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.123164    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.132432 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.132402    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"]
Oct 30 00:42:00.142124 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.142100    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pr5qw\"/\"openshift-service-ca.crt\""
Oct 30 00:42:00.142124 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.142105    4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pr5qw\"/\"kube-root-ca.crt\""
Oct 30 00:42:00.142293 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.142165    4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pr5qw\"/\"default-dockercfg-ms8r5\""
Oct 30 00:42:00.157398 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.157368    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v767j\" (UniqueName: \"kubernetes.io/projected/e54418ca-4007-48a4-b715-f976cb135a45-kube-api-access-v767j\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.157566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.157409    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e54418ca-4007-48a4-b715-f976cb135a45-lib-modules\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.157566 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.157495    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e54418ca-4007-48a4-b715-f976cb135a45-sys\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.157741 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.157583    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e54418ca-4007-48a4-b715-f976cb135a45-proc\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.157741 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.157634    4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e54418ca-4007-48a4-b715-f976cb135a45-podres\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.258130 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.258098    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e54418ca-4007-48a4-b715-f976cb135a45-proc\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.258251 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.258146    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e54418ca-4007-48a4-b715-f976cb135a45-podres\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.258251 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.258181    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e54418ca-4007-48a4-b715-f976cb135a45-proc\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.258251 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.258201    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v767j\" (UniqueName: \"kubernetes.io/projected/e54418ca-4007-48a4-b715-f976cb135a45-kube-api-access-v767j\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.258251 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.258228    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e54418ca-4007-48a4-b715-f976cb135a45-lib-modules\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.258531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.258270    4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e54418ca-4007-48a4-b715-f976cb135a45-sys\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.258531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.258273    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e54418ca-4007-48a4-b715-f976cb135a45-podres\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.258531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.258328    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e54418ca-4007-48a4-b715-f976cb135a45-lib-modules\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.258531 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.258338    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e54418ca-4007-48a4-b715-f976cb135a45-sys\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.280517 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.280490    4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v767j\" (UniqueName: \"kubernetes.io/projected/e54418ca-4007-48a4-b715-f976cb135a45-kube-api-access-v767j\") pod \"perf-node-gather-daemonset-b7hs4\" (UID: \"e54418ca-4007-48a4-b715-f976cb135a45\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.436544 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.436470    4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:00.740329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.740295    4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"]
Oct 30 00:42:00.743581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.743544    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7k5ps_5179b345-9698-4fb0-9628-fc487b250f56/kube-state-metrics/0.log"
Oct 30 00:42:00.745418 ip-10-0-137-69 kubenswrapper[4441]: W1030 00:42:00.745394    4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode54418ca_4007_48a4_b715_f976cb135a45.slice/crio-a2bf43141469367d2e5e6ffd46b4ebf79369fcd10c0bb40b83693fa1d94dec76 WatchSource:0}: Error finding container a2bf43141469367d2e5e6ffd46b4ebf79369fcd10c0bb40b83693fa1d94dec76: Status 404 returned error can't find the container with id a2bf43141469367d2e5e6ffd46b4ebf79369fcd10c0bb40b83693fa1d94dec76
Oct 30 00:42:00.784390 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.784354    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9lbf8_5ba56bd4-0351-41df-9717-c0ba2ec2d124/manager/0.log"
Oct 30 00:42:00.858757 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.858732    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7k5ps_5179b345-9698-4fb0-9628-fc487b250f56/kube-rbac-proxy-main/0.log"
Oct 30 00:42:00.981198 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:00.981172    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7k5ps_5179b345-9698-4fb0-9628-fc487b250f56/kube-rbac-proxy-self/0.log"
Oct 30 00:42:01.166346 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:01.166256    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4" event={"ID":"e54418ca-4007-48a4-b715-f976cb135a45","Type":"ContainerStarted","Data":"797aa3561cd65a56ea5abd98955a74e76792f109a3013eb9cd26ea4767f1738a"}
Oct 30 00:42:01.166346 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:01.166335    4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4" event={"ID":"e54418ca-4007-48a4-b715-f976cb135a45","Type":"ContainerStarted","Data":"a2bf43141469367d2e5e6ffd46b4ebf79369fcd10c0bb40b83693fa1d94dec76"}
Oct 30 00:42:01.166928 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:01.166434    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:01.206485 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:01.206442    4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4" podStartSLOduration=1.206428209 podStartE2EDuration="1.206428209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:01.205141485 +0000 UTC m=+2230.730435388" watchObservedRunningTime="2026-10-30 00:42:01.206428209 +0000 UTC m=+2230.731722085"
Oct 30 00:42:01.320659 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:01.320624    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9lbf8_5ba56bd4-0351-41df-9717-c0ba2ec2d124/manager/0.log"
Oct 30 00:42:02.070784 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:02.070753    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-swwfx_44247589-aae2-437c-807f-4f36d318b7ca/node-exporter/0.log"
Oct 30 00:42:02.184882 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:02.184844    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-swwfx_44247589-aae2-437c-807f-4f36d318b7ca/kube-rbac-proxy/0.log"
Oct 30 00:42:02.256809 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:02.256782    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/1.log"
Oct 30 00:42:02.257097 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:02.257085    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/0.log"
Oct 30 00:42:02.299879 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:02.299858    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-swwfx_44247589-aae2-437c-807f-4f36d318b7ca/init-textfile/0.log"
Oct 30 00:42:02.785179 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:02.785117    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e9581a9-4681-4cfa-ae1a-bc832915c956/prometheus/0.log"
Oct 30 00:42:02.902424 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:02.902393    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e9581a9-4681-4cfa-ae1a-bc832915c956/config-reloader/0.log"
Oct 30 00:42:03.019508 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:03.019484    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e9581a9-4681-4cfa-ae1a-bc832915c956/thanos-sidecar/0.log"
Oct 30 00:42:03.138727 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:03.138627    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e9581a9-4681-4cfa-ae1a-bc832915c956/kube-rbac-proxy-web/0.log"
Oct 30 00:42:03.255353 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:03.255311    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e9581a9-4681-4cfa-ae1a-bc832915c956/kube-rbac-proxy/0.log"
Oct 30 00:42:03.370998 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:03.370965    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e9581a9-4681-4cfa-ae1a-bc832915c956/kube-rbac-proxy-thanos/0.log"
Oct 30 00:42:03.488272 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:03.488245    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e9581a9-4681-4cfa-ae1a-bc832915c956/init-config-reloader/0.log"
Oct 30 00:42:03.908838 ip-10-0-137-69 kubenswrapper[4441]: E1030 00:42:03.908749    4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xvj9n" containerName="extractor"
Oct 30 00:42:04.411059 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:04.411012    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9lbf8_5ba56bd4-0351-41df-9717-c0ba2ec2d124/manager/0.log"
Oct 30 00:42:05.379647 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:05.379613    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l2c22_04666c76-7d71-480d-bae1-27c724358c54/discovery/0.log"
Oct 30 00:42:06.032520 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:06.032494    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/1.log"
Oct 30 00:42:06.032926 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:06.032913    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-877hw_88fc9fb9-507f-455a-93b1-6d158cc5896f/manager/0.log"
Oct 30 00:42:07.018881 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:07.018846    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d88dd7bf-bblqz_36ed0996-1da5-471c-a110-fb73de534126/kube-auth-proxy/0.log"
Oct 30 00:42:07.181119 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:07.181092    4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-b7hs4"
Oct 30 00:42:07.255444 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:07.255417    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-mjckv_972bef50-60cd-4b01-85ab-c17359d1327b/istio-proxy/0.log"
Oct 30 00:42:08.015214 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:08.015182    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9lbf8_5ba56bd4-0351-41df-9717-c0ba2ec2d124/manager/0.log"
Oct 30 00:42:10.960821 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:10.960787    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-748768979-s8jhn_fa78720d-333c-4f7a-891f-1d6e4f120472/authorino/0.log"
Oct 30 00:42:10.998169 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:10.998145    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-t2mxs_03e20d41-ae7d-4ea8-8528-12f00023b3b4/manager/0.log"
Oct 30 00:42:11.341313 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:11.341280    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tmkmd_185b11f8-5a8a-4282-b9ae-53c9182d0668/manager/1.log"
Oct 30 00:42:11.380076 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:11.380046    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tmkmd_185b11f8-5a8a-4282-b9ae-53c9182d0668/manager/0.log"
Oct 30 00:42:11.413310 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:11.413280    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-j8jsw_7883212e-1f9e-471c-be5d-51d0f0b20188/limitador/0.log"
Oct 30 00:42:12.671989 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:12.671948    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l2c22_04666c76-7d71-480d-bae1-27c724358c54/discovery/0.log"
Oct 30 00:42:13.476858 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:13.476822    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xzfnp_3f2407f8-44bd-4a86-af6d-66429fb7ecfd/dns/0.log"
Oct 30 00:42:13.593338 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:13.593298    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xzfnp_3f2407f8-44bd-4a86-af6d-66429fb7ecfd/kube-rbac-proxy/0.log"
Oct 30 00:42:13.957511 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:13.957488    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jvrpq_41e90ea1-6a7c-4007-89f4-bba585722b8a/dns-node-resolver/0.log"
Oct 30 00:42:14.192581 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:14.192556    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d88dd7bf-bblqz_36ed0996-1da5-471c-a110-fb73de534126/kube-auth-proxy/0.log"
Oct 30 00:42:14.430037 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:14.429995    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-mjckv_972bef50-60cd-4b01-85ab-c17359d1327b/istio-proxy/0.log"
Oct 30 00:42:15.359822 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:15.359784    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-76c8cdc68c-8jdsl_df9d6656-9a1a-445e-9b70-f3a9fad5038c/registry/0.log"
Oct 30 00:42:15.468401 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:15.468375    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6hq6g_40bc2c43-b4d6-47f7-86d3-5db7cdcf7f3f/node-ca/0.log"
Oct 30 00:42:17.022329 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:17.022296    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l2c22_04666c76-7d71-480d-bae1-27c724358c54/discovery/0.log"
Oct 30 00:42:18.552079 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:18.552050    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d88dd7bf-bblqz_36ed0996-1da5-471c-a110-fb73de534126/kube-auth-proxy/0.log"
Oct 30 00:42:18.809044 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:18.808926    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-mjckv_972bef50-60cd-4b01-85ab-c17359d1327b/istio-proxy/0.log"
Oct 30 00:42:19.127849 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:19.127771    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-748768979-s8jhn_fa78720d-333c-4f7a-891f-1d6e4f120472/authorino/0.log"
Oct 30 00:42:19.169866 ip-10-0-137-69 kubenswrapper[4441]: I1030 00:42:19.169841    4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-t2mxs_03e20d41-ae7d-4ea8-8528-12f00023b3b4/manager/0.log"